We have finished processing the backlog of occurrences. Again, apologies for interruptions this caused you today. Please expect a post-mortem next week.
Posted Nov 17, 2022 - 14:57 PST
Update
We are continuing to monitor for any further issues.
Posted Nov 17, 2022 - 14:56 PST
Update
We expect to finish processing the backlog of occurrences by 7 PM PT.
Posted Nov 17, 2022 - 10:58 PST
Update
We are continuing to drain the backlog of occurrences. During this time, you may see out-of-date timestamps in the UI, and it may take time for new items to show up.
We apologize for this outage. Please expect a postmortem next week.
Posted Nov 17, 2022 - 10:35 PST
Monitoring
We are now processing the backlog of occurrences.
Posted Nov 17, 2022 - 09:13 PST
Update
We are nearly done with the fix. We expect to begin processing occurrences in 15 minutes.
Posted Nov 17, 2022 - 08:59 PST
Update
We are continuing to work to resolve the issue. We still expect to be done by 9 AM PT.
Posted Nov 17, 2022 - 07:56 PST
Update
We are currently unable to process incoming occurrences. That is, we are storing them – there is no data loss – but unable to fully process them yet. We are working on a remediation. We expect the issue to be resolved by 9 AM PT. Please expect the next update at 8 AM PT.
Posted Nov 17, 2022 - 07:10 PST
Identified
We are experiencing a processing delay in our occurrence pipeline. We are investigating actively and will post when we have identified the issue.
Posted Nov 17, 2022 - 06:30 PST
This incident affected: Processing pipeline (Core Processing Pipeline).