Service outage

Incident Report for Zonos

Postmortem

What products were affected and what was the impact?

All Zonos Services

Impact: CRITICAL

What timeframe did this issue occur?

Date	Time
Feb. 24, 2022	16:39 UTC - 17:29 UTC

How was the issue detected?

The issue was detected by the entire company noticing services not responding and not performing properly.

What functionality was affected?

All Zonos core products were affected. Landed Cost APIs, Checkout, Extensions, Dashboard etc.

What problems did this cause?

Outages across the board.

What was the resolution of the problem and steps that are being taken for continued follow-up?

The problem seemed to be related to Heroku outages and possibly other DNS issues. We have plans to move more critical entities off of Heroku at this time.

What mitigation solutions will we put in place to prevent this issue from occurring in the future?

Depend less on third party software. Stop the flow of traffic when we find total outages in the future.

Posted Feb 24, 2022 - 10:53 MST

Resolved

We are still investigating the cause of the issue, but all issues appear to be up and running at this time.

Posted Feb 24, 2022 - 10:29 MST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Feb 24, 2022 - 10:21 MST

Update

There are a variety of services down worldwide. We are investigating whether this is related.

Posted Feb 24, 2022 - 09:53 MST

Update

We are continuing to investigate this issue.

Posted Feb 24, 2022 - 09:46 MST

Investigating

We are experiencing a service outage. Our team is currently working to restore service. We apologize for any inconvenience. Checkout users may be affected. We will provide an additional update ASAP.

Posted Feb 24, 2022 - 09:39 MST

This incident affected: Landed Cost (Landed Cost API, Landed Cost API (Legacy)), International Checkout, Quoter, Classify, Dashboard, and Plugins (Shopify Duty Tax, BigCommerce Duty Tax, Magento Duty Tax, Salesforce Duty Tax).