Delays in publishing updates to Pages
Incident Report for Yext
Postmortem

Summary

On January 15, 2021, at 2:38 a.m. ET, an urgent ticket was submitted, indicating that publishes for some Pages clients were timing out. There was an issue with downloading Pages files, which impeded publishing for some sites. During investigation, from the period of 6:52 a.m. ET to 9:49 a.m. ET, a server was rolled back to a previous state, causing a subsequent issue of the Code Editor to stop working for Hitchhikers. Once the root cause was identified, Engineers deployed a fix for the timeouts at 1:08 p.m. ET. After this time, publishes that had previously failed started to succeed during the next publish. No data was lost as a result of this outage.

Root Cause

A bug in the caching layer while fetching Pages content caused data to get cached twice, increasing overall publishing latency and causing some publishes to  time out.

Remediation

The root cause has been fixed and we will be implementing additional telemetry around the caching layer.

Posted Jan 21, 2021 - 15:58 EST

Resolved
This incident has been resolved.
Posted Jan 15, 2021 - 15:18 EST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 15, 2021 - 13:31 EST
Identified
We have identified the source of the issue and will implement a fix immediately.
Posted Jan 15, 2021 - 12:44 EST
Update
We have isolated the issue with pages publishing and are working on a resolution. This also caused an outage to the pages code editor, but that has been resolved. This issue is only affecting some sites.
Posted Jan 15, 2021 - 09:53 EST
Update
We are continuing to investigate the root cause of the delay. Although publishing is sometimes delayed many updates are succeeding.
Posted Jan 15, 2021 - 08:06 EST
Investigating
We are currently investigating publishing delays in the Pages platform. Serving existing content to consumers in unaffected.
Posted Jan 15, 2021 - 06:09 EST
This incident affected: Customer Portal Login.