What happened?
At 19:00 UTC on November 18 2021 we started to receive a small number of notifications of some Noun Project icons disappearing from murals and not being visible in the search bar. During our initial investigations we were unable to reproduce the issue. We continued to receive reports and at 09:40 UTC on November 19 2021 we classified this as a critical incident. The relevant code for displaying icons was retained and no data was lost, however the icons were not being displayed.
Details and corrective actions
At 16:26 UTC we updated the configuration for the Noun Project integration, which restored full service with no data loss.
Summary
This incident resulted in a partial outage for a core service, lasting 7 hours and 44 minutes. MURAL remained operational throughout this incident and members were able to log in and access their data; however, some Noun Project icons were not visible in the search menu or on murals.
What we've done to avoid this happening again
We have resolved the configuration issue and restored full service with the use of icons. We are working on several short- and medium-term initiatives to improve the reliability of this service going forward.