Summary

Following a previous incident on the same day, a database instance at our back-ends in Frankfurt became unavailable. This rendered the back-ends to return 404s that would be subsequently cached at our CDNs. Once the issue was identified, the database was synced with the master and the back-end service was restored. There was a number of websites that required a cache clear in order for the 404s to become stale and get the correct content cached.

Impact

• Editor inaccessible in Frankfurt
• Non cached sites responding with 404s in Frankfurt

Root Causes

Redis slave instance in our VPC In Frankfurt was unresponsive.

Trigger

Master Redis instance was unavailable earlier.

Resolution

 - Manually restart the service.

Action Items

• Alert when the Redis slave is not available or not in sync with master.

Lessons learned

• We need better alerting for Redis availability issues.

Timeline GMT+2

2021/06/28 12:28 CET First support reports

2021/06/28 12:28 CET Engineering engaged on the incident

2021/06/28 12:34 CET Incident announced on status page

2021/06/28 13:31 CET Root cause identified

2021/06/28 13:39 CET Back-end service restored

2021/06/28 13:46 CET Status Page updated

2021/06/28 13:46 CET Affected sites still need manual cache clearing upon reporting

2021/06/28 16:32 CET Incident resolved