Affected
- ResolvedResolved
All affected circuits are now up, and we expect them to remain stable.
We apologise for the disruption caused this afternoon / evening. The issue was twofold; initially starting with a requirement to complete an emergency reload of the core at Telehouse West, followed by a hardware failure with one of the switches.
We are working with the hardware vendor to establish the root cause of the initial disruption, and to source replacement hardware. We will put plans in place regarding scheduling these changes, as well as providing a more detailed RFO early next week. This will be made available on request to your account manager or the support team. - IdentifiedUpdate
The majority of NNI are now restored and we can see circuits back up. There remain 2 NNI which are causing issues which we are working to address.
- IdentifiedIdentified
We are handling a hardware failure situation. We are in the process of actively moving affected NNI across to an alternative switch. We will provide further updates as soon as we can.
Circuits not affected by the hardware failure are now online and should remain stable.
Apologies for the continued disruption this evening. - InvestigatingInvestigating
One of our core devices reached a state of memory exhaustion, as a result we had no choice but to complete an emergency reload. This has now taken place, however upon return, the second switch in our chassis entered a stage of looping. We're sending engineers to Telehouse West to investigate.
- IdentifiedIdentified
We are having to complete an emergency reload of our Telehouse West core. This will impact leased line circuits connected to this POP. We would not take this decision lightly, and it has to be actioned immediately. Further updates to follow.
- InvestigatingInvestigating
We unexpectedly lost around several thousand broadband sessions on two LNSs at Telehouse West due to an issue on the Telehouse West core. These sessions should all have reconnected, or be in the process of doing so, to LNSs at other PoPs within our network. Apologies for the disruption this may cause. We will share further details once understood.