BT capacity - Slough
Incident Report for Simwood
Postmortem

We have posted a post-mortem for this and related incidents on our blog.

Posted Nov 12, 2015 - 16:27 UTC

Resolved
Those circuits back in service are performing normally and have been put back into production. We still await an update from BT and repair of the remaining circuits, but as they are not material from a capacity or performance perspective we are marking this incident resolved.

We will follow up with further information on the remaining circuits and any RFO from BT here over the coming days.

Thanks for your patience.
Posted Nov 07, 2015 - 20:43 UTC
Monitoring
We await a further update from BT but 80% of Slough circuits representing the majority of capacity have now come back up. The remaining circuits are to very lightly used exchanges. We will pass on a formal update as soon as we get one.
Posted Nov 07, 2015 - 15:35 UTC
Update
BT engineers remain on-scene in Ealing and there is no further update. We will will receive our next update at 1500Z but will pass on any information as soon as we get it.
Posted Nov 07, 2015 - 13:33 UTC
Update
BT have now confirmed the cause of this issue *is* Ealing as all affected circuits pass through there.
Posted Nov 07, 2015 - 11:16 UTC
Identified
BT have confirmed that the Ealing fault is "very likely" to be affecting us. They have dispatched engineers and will provide us an update at 1300Z. We will of course post further information here if we have it. Should it prove unrelated we will progress with BT and of course have engineers on standby this side.

To repeat, this does not affect service other than degrading capacity and reducing redundancy. We have multiple points of presence into multiple points on the BT network designed for availability and deal directly with BT (unlike competitors designed for improved economy and using third party networks to reach remote exchanges).
Posted Nov 07, 2015 - 10:58 UTC
Update
BT are investigating a transmission failure in the Ealing area of London which they believe may be related.

To clarify our earlier message, this is NOT service affecting. Our IP network is fine and customers do not need to take any action.
Posted Nov 07, 2015 - 10:33 UTC
Investigating
Our links to numerous BT exchanges are showing as down in Slough. Our equipment is showing no errors and we are making efforts to mobilise BT.

Our network is designed such that traffic from all these exchanges will seamlessly flow into other sites, so this is not service affecting. However, this is a material reduction in capacity and reduces our redundancy should another failure occur elsewhere.

Early indications are that this is a BT fault in the Slough area but we await a formal response.
Posted Nov 07, 2015 - 10:22 UTC