Thursday, November 3rd, 2016

WVLS Network WVLS Core Router Unexpectedly Dropping Packets.

At approximately noon today, the WVLS core router began dropping packets (mostly related to Sierra Application traffic) between computers in BCN libraries and the Sierra Application Server. After multiple reports came in within a relatively short window, we began researching the issue.

Some routers on the WVLS network received firmware updates during our regularly scheduled maintenance window on October 18th. As a reminder, our regularly scheduled maintenance window falls on the 3rd Tuesday of each month, from 9:00pm to midnight and can be found on the WVLS Web Calendar (https://calendar.google.com/calendar/embed?src=r0254542jrd8v2lpg8c0sommlc@group.calendar.google.com&ctz=America/Chicago). Recent experiences with (still seemingly random) configuration changes on some current and former Charter library routers after the Oct. 18th updates led us to investigate the Content Filtering service of our core router. The Content Filtering service is included with a bundled license of several security features but has been disabled for the duration of our use of the router. After some preliminary "blanket" application and network troubleshooting, Chris Heitman quickly discovered the unexpected configuration change and was able to resolve it.

While we have now experienced a few unexpected configuration changes like this post firmware update, we have not experienced recurring changes of the same config item on the same router.