Total Expert has successfully resolved Monday’s issue with imports and email sends which resulted in unscheduled database maintenance and placing background processing on hold. The fix included the creation of a new activity streams table which substantially lowers any risk of recurrence.
While investigating the issue, the team placed a proactive hold on processing, which resulted in queues for imports and emails for 24 hours. The hold on processing was lifted Monday evening and the queues were successfully processed in sequential order.
We are continuing to process activity logs and tasks; this work could range from the next few hours up to two days. Please note that Contact list updates may be delayed until we complete processing the events backlog. Our team is tightly managing this final stage to mitigate impact on the overall performance of the platform.
We apologize for the disruption and thank you for your patience as we worked through the issue. Our team will continue to post updates to the Total Expert status page.
Posted Feb 14, 2023 - 13:24 CST
Monitoring
All Functionally has been restored. Emergency database maintenance has now been completed for everything but Activity Logs and Tasks. Activity Logs and Tasks are being actively restored and will be appearing over the next few hours and days. Contact list updates may be delayed until we finish processing our backlog of events.
We will continue to monitor the progress and provide an update in the morning.
Posted Feb 13, 2023 - 19:42 CST
Update
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Paused services include journeys, imports, and email sending. Some pages may display stale data during this window.
We are still on schedule to have paused services operational by 7pm Central.
Posted Feb 13, 2023 - 17:59 CST
Update
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Paused services include journeys, imports, and email sending. Some pages may display stale data during this window.
We expect to have paused services operational by 7pm Central. We will provide another update in 1 hour.
Posted Feb 13, 2023 - 17:04 CST
Update
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled.
Services impacted include journeys, imports, and email sending. Some pages may display stale data during this window. We will provide another update in 1 hour.
Posted Feb 13, 2023 - 15:56 CST
Update
We are continuing to perform emergency database maintenance. We will provide another update in 1 hour.
Posted Feb 13, 2023 - 14:58 CST
Identified
We are currently performing emergency database maintenance.
Posted Feb 13, 2023 - 14:14 CST
Update
We are preparing to perform emergency database maintenance in order to resolve the reported issue.
Posted Feb 13, 2023 - 13:29 CST
Investigating
Some users may be experiencing an issue where system activities are not populating in the user's activity stream.
Posted Feb 13, 2023 - 13:08 CST
This incident affected: Core Platform (API, Email Messages, Event Processing, Reporting, Web Application).