We would like to share more details about the events that occurred with Memsource between 12:15 CEST and 16:06 CEST on July 2, 2021 which led to a partial performance degradation of the Project Management, Editors, API and File Processing services and what Memsource engineers are doing to prevent these sorts of issues from happening again.
12:15 PM CEST: Initial long running requests alerts arrive, but are automatically resolved.
13:27 PM CEST: We start working on optimizing the application code.
13:30 PM CEST: More response time alerts arrive, indicating larger parts of the system are being affected by a performance degradation.
14:26 PM CEST: We deploy new servers, including first code optimizations.
16:06 PM CEST: New servers with fixes are fully deployed resulting in a significant decrease in overall system load. Incident is considered to be resolved.
Newly released functionality caused an overall increase of load on our servers.
Finally, we want to apologize. We know how critical our services are to your business. Memsource as a whole will do everything to learn from this incident and use it to drive improvements across our services. As with any significant operational issue, Memsource engineers will be working tirelessly over the next coming days and weeks on improving their understanding of the incident and determine how to make changes that improve our services and processes.