Close sincerely apologizes for the interruption of our service. We take the stability of a platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. 

Impact 

All Close systems were disrupted for most customers for 23 minutes between 16:48 and 17:11 UTC on November 9th, 2020.

Root Cause & Resolution

The Close application exhausted the available connections to our backend database, preventing the app from functioning normally. This was due to a setting on our new database proxy being set too low to accommodate peak traffic.

Due to a combination of routine internal operations and traffic volume on Monday morning, our application stack consumed the maximum connection limits that were configured on the database proxy.  At 17:00, the problem was diagnosed and we increased the connection limits.  It took approximately another 10 minutes to restore the application back to a completely healthy state.

Timeline

Nov 09 16:48 UTC - Close begins responding to reports of application instability
Nov 09 17:00 UTC - The misconfigured setting on our database proxy is identified and corrected
Nov 09 17:11 UTC - Close application behavior returns to normal.

Next Steps

• Review the best practices for our new database proxy, adjusting the Close app to leverage them
• Make adjustments to how the Close app manages the lifecycle of database connections