On 3/23/2022, a bad deployment to production services caused Drive public APIs (Estimate, Validation, Create Delivery) to fail at from approximately 12:48 PM - 1:10 PM (22 minutes). During this period, DoorDash Drive Merchants were unable to generate quotes for potential deliveries and the Drive system was unable to accept new order fulfillment requests. The issue was mitigated through the rollback of the offending change.
Action taken:
Prevention: We’ve added steps in our staged rollout process to catch these issues before they go to production and impact all merchants.
Detection: We’ve also improved alerting to reduce time to detect of such issues in future
Posted Mar 26, 2022 - 16:50 PDT
Resolved
The issue with Order Creation is now confirmed to be resolved. A bad deployment was discovered and rolled back. Postmortem to follow.
No further updates.
Posted Mar 23, 2022 - 13:14 PDT
Monitoring
The issue with Order Creation is resolved but we will continue to monitor.
Next update in 15 minutes.
Posted Mar 23, 2022 - 13:11 PDT
Investigating
We have detected an issue with Order Creation. We are actively investigating.