BCN NMC added new routes to the WVLS BCN WAN prior to the network being ready for these changes. This caused minor disruption of services. The primary service was DHCP, which would have led to computers losing connectivity incrementally if not addressed soon enough. The issue was caught by the LEAN WI Network Management System (NMS) and Gus Falkenberg raised awareness to WVLS staff immediately. We were able to work with the BCN NMC to rollback the routing adds and resolve connectivity issues.

Update - 10/31/16: It has been discovered that the content filtering services were unexpectedly enabled on routers with comprehensive gateway security licensing (those currently or recently using Charter). SonicWall Support Engineers were able to help Chris discover this unexpected state, and trace some of the Sierra Connectivity issues to it. These have been turned back off as intended.

While inspecting the Minocqua router, it began dropping responses from the DHCP server. This was unexpected behavior. SonicWall Support Engineers advised the creation of a dedicated routing policy for forwarding traffic to the DHCP server. While this solution (though highly specific) appears to be a redundant configuration, and doesn't explain the rule-based packet dropping which had no triggers to match, it does appear to be working. This issue only began affecting Minocqua today (10/31/16), and may have been catalyzed by router inspection. We will remain mindful of this in the off-chance that other routers with similar configurations somehow fall into the same state.

We will monitor the traffic during the course of this evening and when affected libraries come online tomorrow (11/1/16).

Over the past 5-7 business days, we've received an uptick in reports from a few libraries of an increase in error messages referencing JAR file issues and error messages indicating that Sierra licenses are unavailable.

We are investigating these reports to determine the extent of libraries affected and the root cause.