Reason for Outage (RFO) - 2016-07-29
The following is a summary of events for the outage that occurred on the 29th of July 2016.
At 15:00 Friday afternoon, high packet loss at two of our data-centres was detected by our automatic monitoring systems. Immediately following the alert, our data-centre partners were contacted. Throughout the outage, we remained in contact with our partners.
Over the period of an hour packet loss continued to be observed at the two locations. For customers the impact of this outage ranged from being unable to make any calls to experiencing call quality issues. Following the outage, we were informed by our data-centre partners that the outage was the result of a distributed denial-of-service attack aimed at their network. While we were not the target of the attack, the volume of traffic was such as to cause disruption to other businesses making use of their network.
At 16:15, due to preventative measures implemented by our data centre partner, packet-loss subsided completely. Some residual issues were observed after this period - such as delays processing faxes. However, our core service remained stable.
We do not expect further disruption to our services at this time or in the immediate future. However, we have engineers on call 24/7 to diagnose network issues, co-operate with data-centre partners, and aid them in any way possible to find a speedy resolution if the worst does happen.
While this issue was outside our direct control and targeted at the network level of the data centre provider, we are currently working on a solution that will enable us to quickly move all services to another data centre provider should such an attack occur in the future. This solution will be put in place in the coming months.