Summary

Starting on July 8th, 2021, at approximately 8:50pm ET, a small number of Pages sites using reverse proxies began returning error codes for consumer-facing requests. Publishing of content was unaffected. All sites were restored by July 10th at 12:00pm ET.

Root Cause

A change in the IP addresses of Yext's Pages serving infrastructure occurred on July 8th. This change was not picked up by the reverse proxies for some sites, so they continued to send requests to the old IP addresses, resulting in an error.

The affected customers were contacted and assisted in refreshing the DNS caches for their proxies, restoring service.

Remediation

We have requested that affected clients update their proxies to make use of the DNS TTL on our infrastructure to pick up any future IP changes, and our documentation will be updated accordingly.

Posted Jul 19, 2021 - 09:02 EDT

Resolved

This incident has been resolved.

Posted Jul 14, 2021 - 10:41 EDT

Monitoring

The sites that have been identified as affected by this incident are confirmed to be serving requests successfully again, we will continue to monitor the situation.

If your site is working well currently, then it is not affected by this incident. If you are seeing an "Error 1001" page mentioning Cloudflare for all pages on your site hosted by the Yext Pages platform, then your reverse proxy infrastructure is ignoring DNS TTL settings, and you will need to manually flush any DNS caches in your reverse proxy infrastructure.

Posted Jul 10, 2021 - 15:44 EDT

Identified

The root cause of this issue has been identified as a change in the IP addresses of Yext's Pages serving infrastructure last night (US Eastern Time). The only sites affected should be those implemented as a reverse proxy, where requests to Yext Pages are intermediated by some of a Yext client's own infrastructure.

If your site is working well currently, then it is not affected by this incident. If you are seeing an "Error 1001" page mentioning Cloudflare for all pages on your site hosted by the Yext Pages platform, then your reverse proxy infrastructure is ignoring DNS TTL settings, and you will need to manually flush any DNS caches in your reverse proxy infrastructure.

Posted Jul 09, 2021 - 15:17 EDT

Investigating

Unfortunately the fail over did not resolve the problem as expected. We are continuing to investigate.

Posted Jul 09, 2021 - 13:40 EDT

Identified

The issue has been identified and we've failed over to our backup CDN to allow all sites to serve.

Posted Jul 09, 2021 - 13:28 EDT

Investigating

We are currently investigating an outage for a small number of Pages sites. Engineers are applying fixes to affected sites individually while investigating.

Posted Jul 09, 2021 - 13:21 EDT

This incident affected: Pages (Pages Serving).