We've seen no further issues with data ingestion, so we're closing this incident. Thank you for your patience and we wish you a good weekend.
Posted Nov 06, 2020 - 13:34 PST
Monitoring
Data ingestion lag is now gone for all customers. If you see any unusual behavior, don't hesitate to reach out to us at support@iterable.com. Thanks for your patience!
Posted Nov 06, 2020 - 11:05 PST
Identified
This morning at around 7:45am Pacific one of our server clusters experienced a series of crashes due to a malformed query. We have taken steps to disable that query but it has left a significant backlog of data ingestion for that cluster to process. That backlog currently stands at about two hours, although we expect it to decrease significantly throughout the morning.
Note that this only impacts one cluster, so only a small percentage of customers will be affected.
We'll send another update out by 11:30am Pacific Time.