We are still in the phase of fully recovering our services but we wanted to provide you with an update. A more detailed and thorough explanation will be provided through the standard RCA process when it is completed.
At 10:36am EST on December 7, 2021, the Coveo team became aware of issues with a third-party cloud services provider that had impacted multiple Coveo services.
We were able to take mitigating measures during the degradation of the US East Region by redirecting user traffic through alternate regions. This is a temporary measure but it allows search, recommendation and event tracking to operate. Indexing and model building are recovering with some delays.
The provider has communicated that they have taken mitigation actions that show significant recovery in the impacted region. They expect to continue to see improved performance, but do not have an ETA for full recovery at this time.
We are managing the situation and we will keep you informed as it evolves.
Thank you
Posted Dec 07, 2021 - 19:24 EST
Update
We're seeing signs of recovery from our infrastructure provider and we are closely monitoring the impacted services.
Posted Dec 07, 2021 - 17:49 EST
Update
Coveo ML - Models Generator service is back operational.
Posted Dec 07, 2021 - 16:46 EST
Update
We're still working on mitigation for our service which are still impacted.
Posted Dec 07, 2021 - 16:08 EST
Update
The mitigation has been deployed and we're seeing recovery of the search, query suggest and our admin console.
Posted Dec 07, 2021 - 15:02 EST
Update
We're not done yet but are making progress with our mitigation, a subset of requests work.
Posted Dec 07, 2021 - 14:52 EST
Update
Our Load Balancers are affected for the past few minutes, preventing most requests from reaching our platforms.
This affects the Search as well.
We're are investigating.
Posted Dec 07, 2021 - 14:16 EST
Update
We've put workarounds in place and the Analytics Read and Write APIs are back to being operational. However, there are delays processing the Analytics events.
Posted Dec 07, 2021 - 14:08 EST
Update
We're working on a workaround to bring back up the Analytics Write service.
Posted Dec 07, 2021 - 13:25 EST
Update
We've updated the status of our components.
Again, still in talk with AWS but they have no ETA to share yet.
Posted Dec 07, 2021 - 12:34 EST
Update
Our Infrastructure Provider has identified the issue and is actively working on addressing it but has no ETA to share at the moment.
Posted Dec 07, 2021 - 12:00 EST
Update
We updated the status of Analytics Write API to Major Outage.
We're still in talk with our Infrastructure Provider but have no ETA to share as of now.
Posted Dec 07, 2021 - 11:15 EST
Identified
Some of our services are impacted by an AWS issue - Crawling & Push API is delayed - Analytics events write is delayed - Models Building is delayed
Search & models querying are not impacted.
We're in talk with our provider and will post regular updates.
If you need help or to get in touch with us, please visit our Help Portal