Optimizely Search & Navigation (formerly Find) is a cloud-based enterprise search solution that delivers enhanced relevance and powerful search functionality to websites. Between August 9, 2021 and August 12, 2021 we experienced events which impacted the functionality of the service in the EU Digital Experience Cloud region. Details of the incident are described below.
Between August 9, 2021 02:44 UTC and August 12, 2021 08:30 UTC a limited subset of clients using Search & Navigation clusters EMEA13 and EMEA14 experienced intermittent service degradation.
The problem occurred during a capacity planning process, and the source of the malfunction was determined to be human error. As soon as the root cause was identified, appropriate actions were taken to mitigate the impact. The services was fully operational at August 12, 2021 08:30 UTC.
August 9, 2021
02:44 UTC – First alert triggered and troubleshooting started.
03:10 UTC – Issue was identified and workaround initiated.
06:15 UTC – Functionality impacts were assessed, and mitigating plans were devised.
08:08 UTC – STATUS PAGE updated
_08:10 UTC – Long term mitigation plan started.
_
August 12, 2021
08:30 UTC – Mitigation actions completed. Service operational and Incident closed.
The incident was caused by human error during a standard capacity operation which resulted in data inconsistency on the platform.
During the events, a limited subset of client may have experienced network timeout or an inactive service error message.
Short-term mitigation
Long-term mitigation
We apologize for the impact to affected customers. We have a strong commitment to deliver high availability for our Search & Navigation service. We will continue to prioritize our efforts in proving to overcome these recent difficulties, and will do everything we can to learn from the event to avoid a recurrence in the future.