Sunday 19 February 2012
A P1 Incident commenced at 8.36am when our 24/7 Monitoring Platform detected issues with multiple network services across SEQ from multiple upstream carriers (at least Telstra, Allegro, PIPE & iiNet/Westnet were initially affected).
Mach immediately escalated to their respective HelpDesk(s) and Mach was advised that a major point of interconnect (POI – where multiple telco’s integrate services) was “out” due to power failures and this was affecting 3x major Carriers – which meant several redundant and automatic fail-over network path options were not able to function.
Mach then reconfigured its network utilising available services to minimise the affect on our hosting Customers, but some services at 1x of our sites (COY) remained affected and some customer premises sites we monitor remained affected.
An ETA of restoration by 2pm was given to Mach for the upstream carriers to resolve with their power supply vendor. At 11.38am traffic begun flowing again intermittently via upstream carriers and full services restoration was completed by 1.40pm.