Top Menu

Saturday 13 Nov 2010 – Telco Electricity Failure

Saturday 13 November 2010

Electricity failures in a major Brisbane telecommunications hub (Riparian Tower) affects multiple carriers as UPS/Genset/Mains supplies are reconfigured – unplanned outage of services experienced (non-HA services offline during process).

Background

Early this morning, electricity failures in a major Brisbane telecommunications hub (Riparian Tower) affected multiple carriers as UPS/Genset/Mains supplies were reconfigured – unplanned outage of services experienced (non-HA services offline during process); there were no planned maintenance activities at the time.

Issue

Early Saturday morning from 2.43am until 6.56am – for as yet unknown reasons – electricity failures occurred  in a major Brisbane telecommunications hub (Riparian Tower) that affected multiple carriers, and this in turn affected some Mach services as we rely upon some upstream comms services that route/converge/peer via this facility.

There was no issue with any Mach infrastructure; our hosted services were affected by upstream factors completely out of our control.

Resolution Plan

Mach engineers were automatically notified by our 24/7 monitoring system. They immediately escalated to upstream providers, who are focusing on permanent resolution as a Severity 1 issue.

Emergency electricity work is scheduled for early Sunday morning, to resolve root cause – and further interruptions to services may arise when this occurs. Mach has senior Management and Technical resources assigned to monitor all impacts on our services and ensure restoration to normal capabilities with the minimum possible flow-on disruption to our customers.

Status Updates

Mach Technology apologises for the inconvenience caused and will ensure a restoration of services as soon as possible. We will publish further updates as new news comes to hand below…..

  • Update 6.56am: upstream carrier workarounds have restored services that were affecting Mach
  • Update: 9.30am: Mach advised that whilst building electricity services are currently operating in a work-around fashion, emergency electrical work is scheduled for midnight (0001 Sunday 14 November) to resolve root cause – unknown if this will result in further service disruption
  • Update 10.30am: outage response procedures in swing and SLA contract customers advised accordingly
  • Sunday 12.20am: root cause resolution works begin, Mach advised 10 minute interruption will occur – Mach impacted services offline
  • Sunday 1.05am: PIPE peering and other major interconnect services restored
  • Sunday 2.15am: residual services restored via remaining circuits – all Mach impacted services now online
  • Sunday 3.10am: all secondary and tertiary services restored to normal operations
Comments are closed.