We have changed our approach and will not be performing more maintenance to this tonight. It is now stable and a formal maintenance window will be set for next weekend to resolve finally.
Thank you for you understanding during this unplanned work. It was not service affecting and we have had zero support tickets.
Posted Mar 02, 2019 - 20:38 UTC
Update
Please see update below. This post changes operational status only.
Posted Mar 02, 2019 - 15:36 UTC
Update
We have made good progress with this but will be pausing now and resuming later tonight when there are fewer CDRs in flight. The portal and API are no longer considered under maintenance but will be again from 10pm tonight. We will not send notifications then as this is not service affecting.
Posted Mar 02, 2019 - 15:32 UTC
Identified
Overnight we had an outage on a MySQL master. This was non-service affecting as the entirety of call routing does not depend on this, and other functions such as the API read from slaves. The impact was delayed CDRs and delays to number provisioning, both of which depend on a write to a master database of record. This was for a short time whilst the service was restored.
We now need to repair the underlying issue fully which will involve restarting the MySQL master and it being unavailable for periods today. Call routing will not be affected but CDRs may be delayed at times, and new number provisioning may be affected. Certain other API end-points may return temporary errors as well. The API & Portal are therefore considered under maintenance.
This work will start immediately and we'll advise when complete.