After further investigation, we see no other impacts to address. All systems appear to be fully operational.
Posted Apr 28, 2023 - 13:09 PDT
Update
As of 19:41 UTC, the Scheduler Queue has returned to baseline. We have confirmed that POS data has been refreshed for all affected POS types (Brink, Toast, Cloud Connect, Lavu, and Lightspeed), including seeing current day Tickets. Webhooks have resumed as well. With the acute phase of the incident being over, we will check for any other impacts before closing the incident.
Posted Apr 28, 2023 - 12:56 PDT
Monitoring
After scaling up our Scheduler Workers, the queue size has shrunk by ~75%. We will continue to monitor until the queue size is back to baseline.
Posted Apr 28, 2023 - 12:02 PDT
Identified
Around 18:00 UTC, we noticed that our CloudPOS Scheduler queue had an elevated number of tasks waiting to be run. This would likely cause all CloudPOS data to be stale, including Tickets and Clock Entries. It would also lead to delayed webhooks. We are currently scaling up our Scheduler Workers to process the delayed tasks.
Posted Apr 28, 2023 - 11:20 PDT
This incident affected: Core Services (API, Webhooks) and Cloud Integrations (Brink, Aloha Cloud Connect, Toast, Lavu, Lightspeed).
