Incident Report - Disruption to Payments Processing

Incident Date: 2021-09-22

Affected Products: Disbursements, Virtual Accounts, Retail Outlets, Credit Cards, XenPlatform, EWallets, Cash & Balance Reports

Report URL: https://status.xendit.co/incidents/brg12ccwcq8y

What happened? 

[2021-09-22 12:10 GMT+7] An environment variable change was made in one of our core services, however the change also had a hidden side effect which overwrote our web server configuration which caused traffic to no longer be able to reach that core service

[2021-09-22 12:11 GMT+7] We detected an increased rate of failure for that core service which had a cascading effect on services upstream that depended on it

[2021-09-22 12:30 GMT+7] We redeployed the core service which had the effect of reverting the web server configuration back to its previous state

[2021-09-22 12:39 GMT+7] The core service recovered and payments processing was able to resume

[2021-09-22 15:24 GMT+7] Affected payments during the incident time period were fully reconciled and processed.

What measures have we taken to prevent this issue in future?

1. Migrate core services onto an orchestrated application platform with modern secrets and environment variable management which are safer and more reliable. 
2. Extend existing deployment methods to include environment variable changes, so that we can detect similar issues earlier. 
3. Improve automated recovery to minimize disruptions to payments processing in the event of failure in this core system.

‌

We understand that you are counting on our reliability for the smooth operation of your business. We sincerely regret any inconvenience caused to you and your customers. We are committed to do better by applying our learnings from this event to continuously improve our services to serve you better.

If you require any assistance or have further questions, please contact us at help@xendit.co or through live chat at https://www.xendit.co/.

Thank you for your trust in using Xendit to power your business.