Simwood Status

February 2017

BT circuits London
This incident is confirmed resolved. The RFO from BT is not exactly comprehensive and requires further conversations. We will post more in our own RFO. We'd like to thank all customers for their understanding and compliments on how this was handled. Losing 50% of our capacity into/from BT is quite a big deal that could have been a major outage. The way the network is designed ensured it wasn't, and our hierarchical priority of channel allocations (http://blog.simwood.com/2016/07/relaxed-channel-limits/) ensured that all Reserved and Best Efforts capacity was honoured. A proportion of Below Best Efforts traffic (i.e. that above any allocated channel limit which we'd normally let pass if we could) was constrained. We consider this a far better outcome than the alternative.
Feb 24, 11:05-15:43 UTC
Portal
We're still investigating the underlying cause of this however it appears to have been an isolated issue affecting the, off-net hosted, portal connecting to the Simwood API. Customers using the Simwood API directly were unaffected.
Feb 23, 10:36-11:19 UTC
CDR Delays
This incident has been resolved.
Feb 22, 12:19-15:36 UTC
Manchester Site
Things have remained stable so we are marking this resolved. A RFO will follow shortly.
Feb 22, 04:15-14:32 UTC
Google Chrome HTTPS Warnings - portal.simwood.com
The affected certificates have been removed from portal.simwood.com and replaced with an EV SSL certificate. Please accept our apologies for any inconvenience that this may have caused.
Feb 7, 10:25-16:56 UTC

January 2017

Numbers ported from TalkTalk
TalkTalk have confirmed this issue is now resolved, and we are seeing inbound traffic to TalkTalk ported numbers on the Simwood network as expected. As the fault originated outwith the Simwood network we have limited information on the underlying cause but believe that it was due to an issue with a switch on the TalkTalk network which has since been resolved. We apologise for any inconvenience this caused.
Jan 31, 13:56-14:46 UTC
Number Porting
This incident has been resolved.
Jan 12, 13:40-16:30 UTC
Help Centre & Support Tickets
This incident has been resolved.
Jan 12, 13:40-16:30 UTC

December 2016

Manchester<>Slough link
This link has been re-enabled and is passing traffic. We will follow up with the vendor concerned but are marking this incident resolved as inter-site connectivity is again n+1.
Dec 12, 05:37-05:47 UTC

November 2016

[Scheduled] Slough - Third party maintenance
The scheduled maintenance has been completed.
Nov 21, 01:00-07:00 UTC
Simwood Mobile
Regretfully we were unable to satisfy our technical concerns with the preferred solution on the table. They were fundamental and therefore we’ve been unable to proceed. Given our self-imposed time-table we therefore need to call time on our SIM offering for now. For more information, please see our official announcement on the blog; http://blog.simwood.com/2016/11/mobile-provisioning-suspended/ We apologise sincerely for any inconvenience this has caused, but the situation is outwith our control as we know you appreciate.
Nov 1, 19:09 - Nov 18, 17:40 UTC
At Risk - Engineering Works - Manchester
This has now been resolved, without any interruption to the other leg.
Nov 17, 13:17-15:32 UTC
[Scheduled] Manchester - Third party maintenance
The scheduled maintenance has been completed.
Nov 1, 00:00-04:00 UTC

October 2016

Help Centre & Support Tickets
We have received notice that as of 1320 UTC this issue has been resolved. The underlying problem was not at zendesk.com, but rather dyn.com - a major DNS service provider used by Zendesk and many other SaaS providers as well as the likes of AWS, Twitter, Github, Spotify and Heroku - which was subjected to a DDoS attack interrupting many services. For more information see https://www.dynstatus.com/incidents/nlr4yrr162t8
Oct 21, 12:43-13:43 UTC
Number Porting
Porting Requests are now being processed via the Portal and API. Zendesk are still reporting issues, and we have not received notification that the underlying cause has been resolved, so we will continue to monitor this closely.
Oct 21, 12:52-13:36 UTC

September 2016

03 Numbering from Mobiles
We understand that this issue is now resolved. As we were not directly affected, we do not have access to any further information regarding this fault.
Sep 29, 10:30-11:47 UTC
[Scheduled] Essential Maintenance
The scheduled maintenance has been completed.
Sep 24, 22:00-22:08 UTC
Database Server
This issue is now resolved and access to the portal has been restored. Any delayed CDRs should now be available via the Portal and API, please accept our apologies for the delay. As a result of this failure we will be carrying out essential maintenance this weekend that may impact access to the Portal and API. Further information on this will be made available via https://status.simwood.com/ Voice traffic was not affected but we appreciate that many of our customers are heavy API users and we sincerely apologise for interruption to this service.
Sep 22, 17:21-18:06 UTC
Portal CSV CDRs
This issue has been resolved. The CSV CDRs are now in the format described at http://mirror.simwood.com/pdfs/simwood_cdr_format.pdf Please accept our apologies for any inconvenience caused.
Sep 2, 08:56-21:02 UTC

August 2016

[Scheduled] Card Payments and Invoicing
The scheduled maintenance has been completed.
Aug 21, 19:00 - Aug 22, 08:07 UTC
Network Issues / Intermittent Packet Loss
We are confident this issue is resolved, and there has been no recurrence over the last 6 hours, therefore we are now considering this resolved. If you experience any issues please do not hesitate to get in touch.
Aug 10, 09:17-15:11 UTC
Portal Login
This incident has been resolved.
Aug 5, 10:48-11:04 UTC
[Scheduled] [Scheduled] At Risk - Simwood Mobile
These works have been completed without incident.
Aug 4, 20:56-21:25 UTC

July 2016

LONAP packet loss
LONAP have finally replied and confirmed this issue has been resolved. Our port has been brought back into service and looks ok. We're marking this resolved but will be keeping on ongoing eye on it.
Jul 27, 04:41-12:33 UTC
London SIP
This issue has now been resolved, apologies for any inconvenience caused.
Jul 25, 08:21-08:30 UTC
API / Portal
This issue has been resolved as of 1043.
Jul 22, 09:22-12:30 UTC
BT/THN outage - not service affecting
We believe the situation is normalising and are marking this resolved.
Jul 21, 09:01-11:10 UTC
BT outage - non service affecting
We understand this issue has been resolved but are none the wiser as to what the issue actually was within BT's network.
Jul 20, 08:43-12:17 UTC
Portal issues
This has now been rectified.
Jul 14, 09:36-09:38 UTC

June 2016

[Scheduled] At Risk - BT Engineering Works - Slough
The scheduled maintenance has been completed.
Jun 27, 00:15-06:00 UTC
Mobile SIM Configuration
This incident has been resolved.
Jun 20, 10:57-12:08 UTC

May 2016

[Scheduled] DNS changes / CloudFlare
We have seen no issues with this overnight and are therefore marking this work as complete. Thanks again for your understanding.
May 27, 17:40 - May 28, 09:36 UTC
At Risk - BT Engineering Works - London
These works have been completed without incident.
May 27, 10:21-11:22 UTC
Unsubstantiated BT issues
We continue to see or hear no evidence of this incident and are therefore closing this.
May 10, 11:56-13:40 UTC
[Scheduled] Equipment Installation - Slough / London
This work has been completed without incident.
May 5, 14:31-19:34 UTC

April 2016

At Risk - BT Engineering Works - London
BT did not arrive.
Apr 27, 10:04-16:11 UTC
BT outage Slough
BT have now confirmed their fault has been rectified.
Apr 20, 09:49-11:49 UTC
[Scheduled] Third Party Maintenance - Intersite Link
The scheduled maintenance has been completed.
Apr 20, 04:15-06:00 UTC
[Scheduled] Staff Training - Southampton
The scheduled maintenance has been completed.
Apr 14, 09:30-17:01 UTC

March 2016

[Scheduled] Reduced Capacity : BT Engineering Works
The scheduled maintenance has been completed.
Mar 24, 02:00-06:00 UTC
[Scheduled] Third party maintenance - GTT router upgrade
The scheduled maintenance has been completed.
Mar 5, 02:00-06:00 UTC
[Scheduled] Third party maintenance - GTT router upgrade
The scheduled maintenance has been completed.
Mar 4, 02:00-06:00 UTC
[Scheduled] Third party maintenance - Telia cable splicing
The scheduled maintenance has been completed.
Mar 3, 23:00 - Mar 4, 05:00 UTC

February 2016

[Scheduled] Emergency maintenance - Reload of Manchester routers
The scheduled maintenance has been completed.
Feb 18, 21:15-22:08 UTC
Network Issue
The network has been stable for a number of hours now and therefore the Manchester proxy has been re-enabled. The comment below was overly simplistic in categorising affected customers, and the imminent post-mortem will go into greater detail about what occurred and further work required. We apologise again to any customer affected.
Feb 17, 13:23-18:25 UTC
Statistics
This incident has been resolved.
Feb 17, 14:38-18:13 UTC
Inbound Routing Changes
This incident has been resolved.
Feb 16, 16:48-22:06 UTC
Office Telephones
This incident has been resolved.
Feb 4, 16:33-17:35 UTC
Manchester DDoS
This incident has been resolved.
Feb 1, 17:33-18:36 UTC

January 2016

[Scheduled] Manchester - Scheduled maintenance
The scheduled maintenance has been completed.
Jan 28, 18:45-19:37 UTC
Competitor outage
This incident has been resolved.
Jan 13, 13:32-13:35 UTC
EE/O2 Issues
We understand BT have rectified the issue with the new SDIN IPX Platform which was affecting calls between some IPX customers. We do not use IPX and therefore were unaffected by this issue.
Jan 11, 17:37-20:30 UTC
Porting Submission via Portal
This has now been resolved.
Jan 11, 13:22-15:03 UTC

December 2015

Manchester Connectivity
This issue has been resolved, and is being monitored closely. More information to follow.
Dec 24, 12:20-12:51 UTC
[Scheduled] At Risk: SS7 Switch Upgrades - London
The scheduled maintenance has been completed.
Dec 21, 20:03-21:22 UTC
Delays in Operations Desk e-mail responses
Zendesk have reported that the issue causing these delays has been resolved.
Dec 17, 13:15-16:20 UTC
Unable to Submit Port Requests
Submitting ports via the API and Portal should now work as expected. Please accept our apologies for any inconvenience caused.
Dec 17, 13:54-14:21 UTC
[Scheduled] At Risk: SS7 Switch Upgrades - Slough
This maintenance work has been completed as planned.
Dec 14, 21:22 UTC
Incorrect channel limits for some customers
This issue is now resolved, channel usage is once again reflected accurately in the web portal. Apologies for any inconvenience caused, engineers will continue to investigate ways to mitigate future recurrences of this issue.
Dec 9, 15:36-17:53 UTC
[Scheduled] At Risk: SS7 Switch Upgrades - Slough
Unfortunately this maintenance has not taken place and will be rescheduled.
Dec 7, 20:42 UTC

November 2015

[Scheduled] At Risk: SS7 Switch Upgrades - London
The aforementioned works have been completed without incident. Engineers will continuing to monitor the London SS7 switching infrastructure closely but testing has shown normal service restored.
Nov 23, 20:00 - Nov 24, 00:13 UTC
Performance issues with Portal & API
This incident has been resolved.
Nov 23, 16:38-20:30 UTC
Telecity Sovereign House issue does not affect us
There is wide-spread disruption this afternoon owing to what we understand to be a serious of power outages across several floors (and both A & B feeds) of Telecity Sovereign House. This has directly affected those in the facility and has caused flaps of peers across both LINX and LonAP. Given the press is reporting "VoIP outages" we wanted to clarify that Simwood was and remains unaffected by this incident.
Nov 17, 16:12 UTC
Slough BT emergency maintenance
This chassis has been swapped out and internal limits lifted. We had a total of 0 support tickets during this window so thank customers for your patience and trust our handling of it avoided any issues for you. Unfortunately however, this hasn't made any difference to the two exchanges which remain out of service so BT need to investigate further. As we mentioned on Saturday though, these are very light traffic sites and whilst their absence reduces redundancy for incoming calls from some parts of London, only does so to the level most competitors operate at normally. We will issue a post-mortem on this incident in due course as whilst we feel it validates our approach, we're hard on ourselves and there are inevitably lessons to learn.
Nov 9, 13:24-16:02 UTC
International calling
Some customers may have experienced some calls to international destinations being rejected for a short time this morning. During Saturday's BT outage, we adjusted some internal channel limits to reflect the reduced capacity to affected BT exchanges. Unfortunately, due to human error, these were not fully removed and kicked in as weekday volumes came on. Thus international calls which would normally flow out to BT over those exchanges were artificially rejected. This was corrected immediately on us noticing. Apologies for any problems caused.
Nov 9, 10:58 UTC
BT capacity - Slough
Those circuits back in service are performing normally and have been put back into production. We still await an update from BT and repair of the remaining circuits, but as they are not material from a capacity or performance perspective we are marking this incident resolved. We will follow up with further information on the remaining circuits and any RFO from BT here over the coming days. Thanks for your patience.
Nov 7, 10:22-20:43 UTC

October 2015

Provisioning outage for mobile SIMs
This issue is now resolved and provisioning is available on all Simwood SIMs.
Oct 16, 13:23-13:31 UTC

September 2015

Issues with Simwood Portal
This incident has been resolved.
Sep 22, 09:03-22:15 UTC
Intermittent incoming call failures
This incident has been resolved.
Sep 10, 15:35-15:59 UTC
CDR Delays
This incident has been resolved.
Sep 1, 14:24-17:34 UTC

August 2015

[Scheduled] Mobile Provisioning Migration Window
The scheduled maintenance has been completed.
Aug 24, 13:00 - Aug 25, 08:00 UTC
Intermittent outbound call failures via BT network
This issue has now been resolved, engineers will continue to monitor the situation. We will also publish a post-mortem shortly
Aug 13, 09:44-11:59 UTC
[Scheduled] At Risk Notification for 7th August 2015: BT Interconnect
The scheduled maintenance has been completed.
Aug 7, 00:00-08:00 UTC
CDR Delays
This incident has been resolved.
Aug 4, 15:24 - Aug 5, 10:13 UTC

July 2015

Portal Issues
Normal service has, once again, been restored to the Portal. We will continue to monitor the service.
Jul 27, 15:23-17:26 UTC
Portal Speed Issues
Normal service has resumed on the portal, apologies for any inconvenience.
Jul 27, 15:00-15:13 UTC
[Scheduled] Simwood Portal
This work has now been completed and the portal has been returned to service. Please accept our apologies for any inconvenience caused.
Jul 25, 00:17 UTC
Portal unavailable
This incident has been resolved.
Jul 20, 11:13-11:24 UTC
Manchester Network Issue
The issue has been narrowed down to one of our three diverse links between London and Manchester. One of the links has developed packet loss and has now been taken out of service. The issue is no longer service affecting.
Jul 10, 11:00-11:16 UTC
Simwood Mobile terminated traffic failing
This issue is now resolved - we have narrowed it down to a configuration issue with our RAN partner. This issue will be investigated by both parties and we will require a full explanation as to why changes were made without us being notified; although this is a non-production beta product this kind of outage is wholly unacceptable.
Jul 1, 15:06-15:59 UTC

June 2015

Third Party Connectivity Issues
We've now re-enabled Level3 but will continue to monitor.
Jun 12, 09:59-12:51 UTC
Mobile Developer Pack Services Unavailable
Mobile service has now been fully restored and tested. We are now working with our suppliers to complete the order for resilient connectivity and investigating further options to avoid any future outages.
Jun 3, 13:05 - Jun 5, 10:39 UTC