Major downtime

Incident Report for Batch

Postmortem

Here are more details on the impact the incident had on push delivery and the rest of our services from 7:26 UTC to 8:28 UTC.

During the incident:

Notifications delivery: Notifications could not be delivered during the incident.
Access to APIs: Our services were not reachable via our APIs (Campaigns and Transactional APIs). As a consequence, requests sent during the incident could not be taken into account.
Dashboard: The dashboard and our website were not available during the incident, but our status page (https://status.batch.com) remained available.
Statistics: push opens that occurred during the incident could not be processed. For iOS/Android only, the SDK will try to send again the information to Batch the next time users open your app.

After the resolution: Our services were restored and push delivery went back to normal as soon as the problem was fixed by our host, at 8:28 UTC.

Notifications delivery:

Pre-existing push campaigns scheduled to send notifications between 7:26 UTC and 7:49 UTC: These campaigns have not been sent. They will have to be duplicated on the dashboard and sent again.
Pre-existing push campaigns scheduled to send notifications between 7:50 UTC and 8:28 UTC: These campaigns have been sent as soon as the service went back to normal around 8:28 UTC.
Push campaigns scheduled after 8:28 UTC have been sent correctly.

Access to APIs:

All the API calls sent during the incident could not be processed by our services.

Therefore, you will have to resend the requests sent during this period if you have not set up a retry on your end, whether you are using our APIs to:

Create campaigns from the Push Campaigns API
Send notifications to ids via the Transactional API
Create/update audiences or send data via our custom audience, custom data or trigger events APIs.
Export requests via the Export API
Review or remove data for specific users via the GDPR API.

Posted Oct 13, 2021 - 10:09 UTC

Resolved

This incident has been resolved.

Posted Oct 13, 2021 - 09:29 UTC

Update

A fix has been issued by OVH and we are currently monitoring the results.

Posted Oct 13, 2021 - 08:31 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Oct 13, 2021 - 08:28 UTC

Update

It seems that our main hosting provider has an ongoing major outage impacting all of our products.

Posted Oct 13, 2021 - 07:30 UTC

Investigating

We are aware of an ongoing major downtime on our infrastructure. We're currently investigating it and will keep you posted.

Posted Oct 13, 2021 - 07:26 UTC

This incident affected: API (Transactional API, Campaigns API, Custom Data API, Transactional API for Partners), 3rd party integrations (Segment, mParticle, Salesforce Marketing Cloud), Data (Exports), Dashboard (Main dashboard, Editorial dashboard), Webservices (SDK, Inbox, Custom attributes, In-app messaging, Web push static resources), and Delivery (Transactional push, Push campaigns, In-app messaging).