Issue: Rules Engine and Data Processing queues blocked
Cause: A power failure at a service provider data center.
Incident Timeline on the 22nd of December:
13:34 UTC - We received notice that our upstream platform service experienced a power outage in a specific availability zone, which had an impact on Gainsight CS as well as other supporting services. While some customers witnessed very little impact, others had queues blocked to prevent data loss. The outage caused corruption that required data to be manually restored for some customers.
18:35 UTC - A subset of customer queues remained in a blocked state as we recovered reporting and dashboard data. Our engineers also added resources to help bring queues current. We maintained an open bridge with vendor support engineers who helped with recovery.
21:59 UTC - By this time, all report data was restored and errors subsided.
Resolution:
While the actual resolution had to do with power restoration at service provider level, we also discovered an opportunity for better data resiliency which we have taken into account to avoid larger scale issues like this.
Thanks for your patience and please email support@gainsight.com with any questions.
Global queues have been unblocked but we are still recovering an isolated set of customers which remained blocked.
We are adding resources to help bring queues current. Please expect delays as we work through backlog.
We continue to work on other recovery actions.
We will update as we have more information.
While the queues are still in a blocked state, we observe the loading of High Volume Reports is also impacted.
Rules Engine and Data Processing queues are currently blocked to prevent any impact on customer jobs.