M365 - Users may be unable to access the admin portal

Resolved
Minor outage
Started over 2 years ago Lasted about 6 hours

Affected

Cloud Services
Microsoft 365
Updates
  • Resolved
    Resolved

    Final status: Final status: Microsoft's actions to re-distribute traffic across the healthy infrastructure have completed and their telemetry systems indicate that impact has been remediated. Additionally, they have worked with some users who were previously impacted and they have confirmed that they are no longer experiencing impact.

    Scope of impact: The majority of the reports originated from customers in Europe, the Middle East, and Africa. However, this could potentially have also affected users across other regions.

    Start time: Thursday, February 3, 2022, at 8:50 AM UTC

    End time: Thursday, February 3, 2022, at 3:50 PM UTC

    Root cause: An issue within caching infrastructure resulted in impact.

    Next steps:

    • Microsoft are analyzing performance data and trends on the caching systems to help prevent this problem from happening again.

    This is the final update for this event.

  • Monitoring
    Update

    Microsoft is continuing their efforts to mitigate the issue by re-distributing user traffic across healthy infrastructure to accelerate recovery. They have also routed some traffic to additional, healthy infrastructure to help with remediation efforts.

    Next update by: Thursday, February 3, 2022, at 5:30 PM UTC

  • Monitoring
    Monitoring

    Microsoft have reverted the update and completed the manual restarts of the affected infrastructure. They are monitoring the status to ensure impact has been fully mitigated. The service health is improving across the affected infrastructure, but the recovery process has been slower than anticipated and they are taking the necessary actions to expedite this.

    Scope of impact: Majority of the reports have originated from customers in Europe, Middle East, and Africa. However, this could potentially also affect users across other regions.

    Next update by: Thursday, February 3, 2022, at 3:30 PM UTC

  • Identified
    Update

    Current status: The revert of the update is taking longer than expected to complete. Microsoft are continuing their efforts to resolve the issue by also manually restarting the affected infrastructure to expedite the recovery.

    Next update by: Thursday, February 3, 2022, at 1:30 PM UTC

  • Identified
    Identified

    Current status: Microsoft have identified a recent service update designed to improve user experience is causing impact. They are reverting the offending update to resolve impact.

    Scope of impact: Impact is specific to users who are are hosted in Europe, Middle East, and Africa.

    Next update by: Thursday, February 3, 2022, at 12:00 PM UTC