All Deployments: Impact to phone call authentication delivery and SMS delays in China and the Netherlands
Postmortem
Netherlands/China Telephony outage - All Deployments
Incident Report - 2020/07/17
Summary:
From 17:12 EDT to 19:01 EDT on July 17, 2020, all Duo deployments experienced VOIP and SMS authentication failures for the Netherlands and China. Failures were due to a widespread DNS outage that affected one of Duo’s telephony providers. The Duo Engineering Team resolved the issue and restored service at 18:16 EDT for the Netherlands. Service to China was restored after our telephony provider implemented a fix.
Details:
At 17:35 EDT on July 17, the Duo Engineering Team was alerted to failing VOIP and SMS authentications.
At 17:39 EDT, the Engineering team determined that failures were due to a widespread DNS outage affecting a large portion of the internet, including one of Duo’s telephony providers.
At 17:57 EDT, the team determined that only telephony service in China and the Netherlands was affected by the DNS outage.
At 18:16 EDT, the team manually failed over to another telephony provider to restore service to the Netherlands.
At 18:53 EDT, our telephony provider reported that they had implemented a fix to restore service.
The Duo Engineering team continues to work with existing and new telephony partners to ensure redundancy and uptime in the future.
Posted Jul 20, 2020 - 14:11 EDT
Resolved
The issue with Phone Call Delivery and SMS delays to China and the Netherlands is fully resolved and all services are now fully functional.
We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.
Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted Jul 17, 2020 - 19:07 EDT
Identified
Our Engineering Team has identified the cause of the Phone delivery and SMS delays on our deployments and are actively working to restore service. Impact has been identified to only affect customers in China and the Netherlands at the moment.
Please check back here or subscribe to updates for any changes.
Posted Jul 17, 2020 - 18:13 EDT
Investigating
We are currently investigating an issue that may cause phone authentication to fail on all of our deployments and are working to correct the issue as soon as possible.
Please check back here or subscribe to updates for any changes.
Posted Jul 17, 2020 - 17:51 EDT
This incident affected: DUO1 (Phone Call Delivery, SMS Message Delivery), DUO2 (Phone Call Delivery, SMS Message Delivery), DUO3 (Phone Call Delivery, SMS Message Delivery), DUO4 (Phone Call Delivery, SMS Message Delivery), DUO5 (Phone Call Delivery, SMS Message Delivery), DUO6 (Phone Call Delivery, SMS Message Delivery), DUO7 (Phone Call Delivery, SMS Message Delivery), DUO8 (Phone Call Delivery, SMS Message Delivery), DUO47 (Phone Call Delivery, SMS Message Delivery), DUO10 (Phone Call Delivery, SMS Message Delivery), DUO11 (Phone Call Delivery, SMS Message Delivery), DUO12 (Phone Call Delivery, SMS Message Delivery), DUO13 (Phone Call Delivery, SMS Message Delivery), DUO14 (Phone Call Delivery, SMS Message Delivery), DUO15 (Phone Call Delivery, SMS Message Delivery), DUO16 (Phone Call Delivery, SMS Message Delivery), DUO17 (Phone Call Delivery, SMS Message Delivery), DUO18 (Phone Call Delivery, SMS Message Delivery), DUO19 (Phone Call Delivery, SMS Message Delivery), DUO20 (Phone Call Delivery, SMS Message Delivery), DUO21 (Phone Call Delivery, SMS Message Delivery), DUO22 (Phone Call Delivery, SMS Message Delivery), DUO23 (Phone Call Delivery, SMS Message Delivery), DUO24 (Phone Call Delivery, SMS Message Delivery), DUO25 (Phone Call Delivery, SMS Message Delivery), DUO26 (Phone Call Delivery, SMS Message Delivery), DUO27 (Phone Call Delivery, SMS Message Delivery), DUO28 (Phone Call Delivery, SMS Message Delivery), DUO29 (Phone Call Delivery, SMS Message Delivery), DUO30 (Phone Call Delivery, SMS Message Delivery), DUO31 (Phone Call Delivery, SMS Message Delivery), DUO32 (Phone Call Delivery, SMS Message Delivery), DUO33 (Phone Call Delivery, SMS Message Delivery), DUO34 (Phone Call Delivery, SMS Message Delivery), DUO36 (Phone Call Delivery, SMS Message Delivery), DUO37 (Phone Call Delivery, SMS Message Delivery), DUO38 (Phone Call Delivery, SMS Message Delivery), DUO39 (Phone Call Delivery, SMS Message Delivery), DUO40 (Phone Call Delivery, SMS Message Delivery), DUO41 (Phone Call Delivery, SMS Message Delivery), DUO42 (Phone Call Delivery, SMS Message Delivery), DUO43 (Phone Call Delivery, SMS Message Delivery), DUO44 (Phone Call Delivery, SMS Message Delivery), DUO45 (Phone Call Delivery, SMS Message Delivery), DUO46 (Phone Call Delivery, SMS Message Delivery), DUO48 (Phone Call Delivery, SMS Message Delivery), DUO9 (Phone Call Delivery, SMS Message Delivery), DUO49 (Phone Call Delivery, SMS Message Delivery), DUO50 (Phone Call Delivery, SMS Message Delivery), DUO51 (Phone Call Delivery, SMS Message Delivery), DUO52 (Phone Call Delivery, SMS Message Delivery), DUO53 (Phone Call Delivery, SMS Message Delivery), DUO54 (Phone Call Delivery, SMS Message Delivery), DUO55 (Phone Call Delivery, SMS Message Delivery), DUO56 (Phone Call Delivery, SMS Message Delivery), DUO57 (Phone Call Delivery, SMS Message Delivery), DUO58 (Phone Call Delivery, SMS Message Delivery), DUO59 (Phone Call Delivery, SMS Message Delivery), DUO60 (Phone Call Delivery, SMS Message Delivery), DUO61 (Phone Call Delivery, SMS Message Delivery), DUO62 (Phone Call Delivery, SMS Message Delivery), DUO63 (Phone Call Delivery, SMS Message Delivery), DUO64 (Phone Call Delivery, SMS Message Delivery), DUO65 (Phone Call Delivery, SMS Message Delivery), and DUO35 (Phone Call Delivery, SMS Message Delivery).