Telehouse West - Connectivity Issues

Resolved
Partial outage
Started over 1 year ago Lasted about 4 hours

Affected

Connectivity & Voice
Inspira Broadband
Inspira Leased Lines & EFM
Updates
  • Resolved
    Resolved

    All affected circuits are now up, and we expect them to remain stable.

    We apologise for the disruption caused this afternoon / evening. The issue was twofold; initially starting with a requirement to complete an emergency reload of the core at Telehouse West, followed by a hardware failure with one of the switches.

    We are working with the hardware vendor to establish the root cause of the initial disruption, and to source replacement hardware. We will put plans in place regarding scheduling these changes, as well as providing a more detailed RFO early next week. This will be made available on request to your account manager or the support team.

  • Identified
    Update

    The majority of NNI are now restored and we can see circuits back up. There remain 2 NNI which are causing issues which we are working to address.

  • Identified
    Identified

    We are handling a hardware failure situation. We are in the process of actively moving affected NNI across to an alternative switch. We will provide further updates as soon as we can.

    Circuits not affected by the hardware failure are now online and should remain stable.

    Apologies for the continued disruption this evening.

  • Investigating
    Investigating

    One of our core devices reached a state of memory exhaustion, as a result we had no choice but to complete an emergency reload. This has now taken place, however upon return, the second switch in our chassis entered a stage of looping. We're sending engineers to Telehouse West to investigate.

  • Identified
    Identified

    We are having to complete an emergency reload of our Telehouse West core. This will impact leased line circuits connected to this POP. We would not take this decision lightly, and it has to be actioned immediately. Further updates to follow.

  • Investigating
    Investigating

    We unexpectedly lost around several thousand broadband sessions on two LNSs at Telehouse West due to an issue on the Telehouse West core. These sessions should all have reconnected, or be in the process of doing so, to LNSs at other PoPs within our network. Apologies for the disruption this may cause. We will share further details once understood.