Top Menu

Wednesday 22 Mar 2017 – Widespread multi-carrier Peering/Packet Loss Issue

Wednesday 22 Mar 2017

Overview

At exactly 7.12am this morning, Mach monitoring systems detected issues concerning multiple carriers that gave packet-loss issues to many sites across (at least) NSW and QLD.

All Mach infrastructure was first ruled out as having no issue nor any recent configuration changes. Evidence was gathered and multiple HPIM (High Priority Incident Management) Tickets have been raised by Mach staff on behalf of several of our clients whose underlying comms link(s) have been affected.

For our clients, this most affected Remote Desktop (RDP) and other services acutely sensitive to packet loss.

The widespread multi-carrier Peering/Packet Loss Issue has affected services the most, between the hours of 7.12am and stabilised somewhat by 9.37am as various low level network paths were reconfigured by multiple carriers, and more so at 9.56am when Mach observed changes VOCUS made with their NSW-QLD interconnects (had been suffering 40% packet loss and 4000ms latency).

Confirming that Mach has proven this issue affects a number of other Telco’s including: TPG, IX, PIPE, AAPT, iiNet and NextGen at sites across NSW and QLD.

Further Details

  • 11am Update:
    • Still not resolved: TPG Private WANs are still affecting two clients, each at geographically separate locations
    • Still not resolved: NextGen Public Internet (1x service) traced back to VOCUS Upstream, that is affecting one Internet service at one Mach Data Centre
    • Four other related areas impact have been resolved
    • Mach remains in contact with each Client as impact pertains to them
  • 1pm Update:
    • Above 2x separate issues, as they impact two separate clients per above remain not fully resolved.
    • Expect to be resolved overnight, and root cause report being obtained from providers/carriers
  • If you having concerns regarding your current network or how this issue impacted you, please Contact Us so that we can assist – including providing alternatives/additional redundancy if necessary.

Clients can see the latest up to date information of Status Notices online at: http://status.mach.com.au/

Comments are closed.