Top Menu

Friday 20 May 2011 – Telco Internet Outage

Friday 20 May 2011

One of Mach’s Data Centre sites (Cooroy) temporarily lost some network (Public Internet) communications for 2hrs due to an upstream Service provider failure, until they resolved their failed maintenance change.

Background

Early this morning, one of Mach’s Data Centre sites (Cooroy) temporarily lost network (Public Internet) communications due to an upstream Service provider failed change during their planned overnight maintenance window. Only clients that did not have a multi-carrier redundant solution were impacted; there were no planned Mach maintenance activities at the time.

Issue

From 7.00am upstream network carrier services from Allegro networks were offline (their BGP Peer services) that adversely affected some services hosted at one of Mach’s four Data Centres. This presented to some users as an inability to connect to hosted servers, email send/receive or display of web pages.

There was no issue with any Mach infrastructure; our hosted services were affected by upstream factors completely out of our control.

Resolution Plan

Mach engineers were automatically notified by our 24/7 monitoring system. Immediate investigation was undertaken to identify root cause of the problem. They then escalated to the Allegro team, who then resolved the issue.

Status Updates

Mach Technology apologises for the inconvenience caused and will ensure a restoration of services as soon as possible. We will publish further updates as new news comes to hand below…..

  • Update 7.00am: outage response procedures in swing and process started re SLA contract customers advised accordingly
  • Update 8.30am: root cause proven and vigorous engagement with Allegro to complete restoration of BGP Peer services and therefore, correct routing of Public Internet traffic to the Cooroy Data Centre site via their comms links
  • Update 9.10am: communications restored to correct level of quality

 

Comments are closed.