At 2023-09-12 6:00 AM WIB we performed a planned maintenance to one of the production clusters. This maintenance activity was planned without downtime expected.
At 2023-09-12 6:35 AM WIB we completed the maintenance but identified the deployments were not in a healthy state and identified degraded performance in our Payments, Payouts, Credit Card, Checkout UI, Subscription, and XenPlatform’s Transfer APIs, resulting in customers getting 500/503 HTTP error response codes.
At 2023-09-12 6:45 AM WIB, we initiated the rollback of the changes, and we began to see issues getting partially resolved.
At 2023-09-12 7:14 AM WIB, we completed the recovery process and resumed processing new incoming requests.
Our investigation revealed an unexpected edge case during the infrastructure maintenance of one of our production infrastructure clusters, causing this outage. The cluster was one of the last two clusters we aimed to upgrade. This edge case was not found during the testing phase and upgrades of other production clusters.
We understand that you are counting on our reliability for the smooth operation of your business. We sincerely regret any inconvenience caused to you and your customers. We are committed to do better by applying our learnings from this event to continuously improve our services to serve you better.
If you require any assistance or have further questions, please contact us at help@xendit.co or through live chat at https://www.xendit.co/.
Thank you for your trust in using Xendit to power your business.