2023-06-13 Outage - Dashboard
Incident Report for Zonos
Postmortem

What products were affected and what was the impact?

Zonos Dashboard

Impact: CRITICAL

 

What timeframe did this issue occur?

Date Time
Jun 13, 2023] 12:54 to 13:46 MDT

 

How was the issue detected?

Internal reports of authorization failures and Dashboard becoming inaccessible.

What functionality was affected?

Zonos Dashboard was not accessible.

What problems did this cause?

Users were unable to access Dashboard to complete tasks.

What was the resolution of the problem and steps that are being taken for continued follow-up?

The issue was identified as an AWS Operational issue in the US-EAST-1 Region impacting an upstream service provider hosting our Front-End services for Dashboard. We were able to redeploy those services to an unaffected region to restore functionality.

What mitigation solutions will we put in place to prevent this issue from occurring in the future?

We are continually assessing and improving business continuity solutions throughout every layer of our tech stack to minimize downtime and automate recovery where possible.

Posted Jun 13, 2023 - 16:55 MDT

Resolved
This incident has been resolved.
Posted Jun 13, 2023 - 13:49 MDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 13, 2023 - 13:46 MDT
Identified
An issue with upstream Lambda creation and execution has been identified, and we are waiting on a fix to be rolled out while investigating other mitigation strategies. For more information, see the AWS status at https://health.aws.amazon.com/health/status.
Posted Jun 13, 2023 - 13:37 MDT
Update
We are continuing to investigate this issue.
Posted Jun 13, 2023 - 13:27 MDT
Investigating
We are currently investigating reports of a potential service interruption with Dashboard. We apologize for any inconvenience and will post another update as soon as we learn more.
Posted Jun 13, 2023 - 13:25 MDT
This incident affected: Dashboard.