This issue was related to one of our Manchester ethernet customers managing to create a loop between two remote office sites. This caused a broadcast storm which was not apparent in our monitoring (it should have been) and resulted in CPU on the access switch stack rising. Whilst all forwarding is performed in hardware on all our network equipment, the CPU is obviously required for routing protocols. The consequence of this therefore was intermittent and eventually lost connectivity up to the redundant routers above them.

The culprit customer's ports were disabled to rectify the immediate problem and then on examination of logs they were reconfigured with elevated loop detection.

All has remained stable since and we apologise for any inconvenience caused. This has highlighted inadequacies in our monitoring which we will address as soon as possible.