Elevated Waitlist Partner APIs Errors

Incident Report for Yelp Developer APIs and Services

Postmortem

On Feb. 22, 2022, during 3:31 am PST until 11:30 pm PST, some partners started experiencing occasional errors and increased latency while using Waitlist Partner APIs.

This incident is mainly caused by the internal infrastructure degradation that affected our service. During this time, our service received 2210 requests. Based on the evaluation, this incident was categorized as a critical impact due to all incoming requests being affected by this incident.

By 06:50 am PST, the root cause has been identified as the internal infrastructure outage that cascading affected the service.

By 08:09 am PST, the fix was implemented to this particular issue and we started seeing the status of our service going back to the normal position.

The actual duration of this incident is between 3:31 am PST until 11:30 pm PST today.

We are truly sorry for the inconvenience and thank you very much for bearing with us! Now Waitlist Partner APIs is fully operational. Please feel free to reach out waitlist-api-support@yelp.com for any concerns or questions.

Thank you!

Posted Feb 22, 2022 - 11:40 PST

Resolved

The incident has been resolved. The total time span of this incident is from 3:31 am PST until 11:30 am PST today.

A postmortem is on its way to provide detailed information about this incident.

Posted Feb 22, 2022 - 11:29 PST

Update

We are continuing to monitor for any further issues.

Posted Feb 22, 2022 - 10:46 PST

Update

We are continuing to monitor for any further issues.

Posted Feb 22, 2022 - 09:41 PST

Update

We are continuing to monitor for any further issues.

Posted Feb 22, 2022 - 09:09 PST

Update

We are continuing to monitor for any further issues.

Posted Feb 22, 2022 - 08:39 PST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Feb 22, 2022 - 08:09 PST

Update

We are continuing to work on a fix for this issue.

Posted Feb 22, 2022 - 07:19 PST

Identified

The issue has been identified and a fix is being implemented.

Posted Feb 22, 2022 - 06:50 PST

Update

We are continuing to investigate and will follow up with more details in 30 minutes.

Posted Feb 22, 2022 - 06:10 PST

Update

Engineering teams are still working through resolution of the underlying issue. We are continuing to investigate and will follow up with more details in 30 minutes.

Posted Feb 22, 2022 - 05:26 PST

Update

The issue appears to be caused by an infrastructure-level outage. Around 50% of requests are failing. We are continuing to investigate

Posted Feb 22, 2022 - 04:50 PST

Investigating

We're experiencing an elevated level of API errors.

We are investigating, and will follow up with more details in 30 minutes.

Posted Feb 22, 2022 - 04:37 PST

This incident affected: Waitlist Partner APIs (Waitlist APIs, On My Way APIs, Loyalty APIs, Reservation APIs).