On October 26, 7:52 UTC we received a purge request containing unparsable data, which caused elevated delays in fulfilling new purge requests. Our engineering team identified the issue, implementing a short term fix to block corrupted purge requests. The purging service completely recovered at 9:48 UTC.
Customers may have seen up to a 1.5 hour delay in purge requests, though no customers reported any issues related to the incident.
During this incident, we found that our internal logs lacked some of the proper data to isolate the issue to the purger. This slowed our initial investigation, though once the issue was identified, our engineers pushed a fix to immediately resume purging operations.
We will be implementing even more stringent input validation of purge requests to prevent re-occurrence of this issue. We will also be updating our troubleshooting processes to speed up root cause analysis of purging issues in the future.