Resolved -
This incident has been resolved.
Jul 13, 18:41 UTC
Monitoring -
We have some out of hours house-keeping to do here but this incident is otherwise resolved. We will be monitoring for a few hours before closing to be certain.
Jul 13, 16:39 UTC
Update -
CDRs are now being processed at full speed and catching up rapidly.
Jul 13, 16:04 UTC
Update -
Portal and API (v4) are now fully operational. CDRs remain lagged.
Jul 13, 15:13 UTC
Identified -
We have lost hardware in Slough.
Voice services have failed over as expected and are unaffected. Most other services have recovered too although we are currently dealing with some Redis master failures elsewhere on the network. This will manifest in portal logins and writing to selected API end-points. Separately CDR processing is several million delayed so API balances etc. will not be current.
We are recovering Redis clusters as P1 and will advise progress.
Jul 13, 14:29 UTC