Friday 9 December 2011
An upstream public internet routing unplanned intermittent outage affected some services into our Cooroy Data Centre. Mach engineers immediately responded when the issue arose at 1.35pm and remained in continuous discussions with upstream providers to ensure their issues were resolved as soon as possible.
We can confirm there was no failure or issue with any Mach services or infrastructure and our multiple redundant links remained 100% up.
The root cause of this P1 declared incident was upstream multi-carrier “peering” and interconnections of BGP routed traffic and was resolved after six intermittent outages between 1.35pm to 4.40pm, i.e.:
- affected comms services were first restored at 1.53pm (18min outage);
- however went offline again at 2.01pm and then came back up at 2.21pm (20min outage);
- they then went offline again (a third time) at 2.26pm and then came back up at 2.45pm (19min outage);
- they then went offline again (a fourth time) at 2.50pm and then came back up at 3.20pm (30min outage);
- they then went offline again (a fifth time) at 3.26pm and then came back up at 3.30pm (4min outage);
- they then went offline again (a sixth time) at 4.10pm and then (after a final 30min outage) returned to normal operation at 4.40pm.