We have changed our approach and will not be performing more maintenance to this tonight. It is now stable and a formal maintenance window will be set for next weekend to resolve finally.
Thank you for you understanding during this unplanned work. It was not service affecting and we have had zero support tickets.
Posted 8 months ago. Mar 02, 2019 - 20:38 UTC
Please see update below. This post changes operational status only.
Posted 8 months ago. Mar 02, 2019 - 15:36 UTC
We have made good progress with this but will be pausing now and resuming later tonight when there are fewer CDRs in flight. The portal and API are no longer considered under maintenance but will be again from 10pm tonight. We will not send notifications then as this is not service affecting.
Posted 8 months ago. Mar 02, 2019 - 15:32 UTC
Overnight we had an outage on a MySQL master. This was non-service affecting as the entirety of call routing does not depend on this, and other functions such as the API read from slaves. The impact was delayed CDRs and delays to number provisioning, both of which depend on a write to a master database of record. This was for a short time whilst the service was restored.
We now need to repair the underlying issue fully which will involve restarting the MySQL master and it being unavailable for periods today. Call routing will not be affected but CDRs may be delayed at times, and new number provisioning may be affected. Certain other API end-points may return temporary errors as well. The API & Portal are therefore considered under maintenance.
This work will start immediately and we'll advise when complete.