Resolved -
This incident is now fully resolved and appears to have been a fault on the Simwood network.
A minor change relating to the validation of caller ID against Ofcom data was rolled around the network from 11am. It had passed all unit tests and presented no errors in production. However, it had the consequence of us assuming at the time of calls being routed that our costs included maximum origin surcharges on destinations that apply them, i.e. UK mobile networks except O2 and certain fixed line operators. While we routinely route calls at a loss and do not pass on origin surcharges, the consequence of this was that customer profiles became progressively loss-making for these destinations and overall. Our fraud and anti-arbitrage protections then began blocking such ostensibly loss-making traffic for customers who had tripped their account or destination specific loss-limits. Surcharges vary between 34ppm and 200ppm depending on network meaning account traffic profile would have affected if and when limits were reached. Some did not trip at all, others would have progressively tripped after 11.08, resulting in some calls to affected destinations being blocked. Calls to destinations without surcharges were unaffected as were those for customers who had not tripped particular limits; nevertheless it resulted in a progressive reduction in our traffic to certain networks over the ensuing hour.
Our first support ticket was at 11:42, with a greater volume of reports from 12:00 and mitigation(s) fully deployed by 12:13. There were two mitigations: 1) customer loss-limits were raised at 12:11 and 2) the deployed image was fully rolled back around the network from 12:10 to 12:13. The last call exhibiting this behaviour, albeit not blocked, was 12:13.
We identified and corrected the bug introduced in logic and are adding further unit tests to ensure such an occurrence cannot repeat. Once the new build fully passes the new tests, this updated image will be carefully deployed later today. We have also added specific global margin monitoring to alert very quickly on unnaturally changes in apparent profitability.
Apologies for this issue which, as usual, we own and will learn from.
Aug 1, 12:28 UTC
Monitoring -
We believe this was mitigated at 12.13 - any examples after that please let us know. We believe we understand what the issue was but continue to investigate and will report fully shortly. To correct the previous report, this would have affected calls to UK destinations applying an origin surcharge - mobile networks are the most egregious but some landlines would have been affected too.
Aug 1, 11:46 UTC
Identified -
We were seeing reduced volume to certain UK mobile networks which we believe we have mitigated. We are investigating why this might be with the last report being at 12.06 BST.
Aug 1, 11:23 UTC