This began impacting ReSci clients as of about 6AM PST. We are seeing delays and incorrect statistics / counts in our analytics dashboards as well as a delay in some morning sends.
Our main Cortex dashboard, SMS, data science APIs, events APIs, template tools, and other APIs remain operational as expected.
Here is an update on the affected components.
Promo Blasts: Operational As of 1PM PST, all promo blasts that were delayed from this morning (from ~6AM PST) have been sent out.
Realistic Test Sends: Degraded performance A small percentage of test-sends are timing out. This is resulting from the system delays we experienced this morning.
Click tracking: Operational
Open Tracking: Degraded performance Some “open” events are missing from analytics and filtering from approximately ~6AM PST until now.
JS Events: Degraded performance Some events are missing from analytics and filtering from approximately ~6AM PST until now.
Segment sync to FB and Google Degraded performance We are experiencing issues with segments being automatically synced to Google and Facebook.
Posted Nov 25, 2020 - 16:22 PST
Monitoring
Although there is no longer an issue with emails sending, the outage with our infrastructure provider is causing delays in analytics for sends since this morning. Sends prior to today should reflect accurate reporting, and once the outage is resolved we will backfill reports to reflect data accurately. JavaScript tracking of onsite events is also impacted and thus page views/carts/checkouts are currently missing and will be backfilled once the outage is resolved. Open and click tracking are working as expected. We expect a full recovery to take a few hours and will continue to monitor and update as we learn more.
Posted Nov 25, 2020 - 14:27 PST
This incident affected: Data Imports Service and Reporting.