Everything is looking good and has been stable the last hour 🎉
Posted 9 years ago. Nov 25, 2016 - 12:39 EST
Update
We've finished processing all check-ins that were left on our failover queue. Everything is looking to be back to normal. We'll continue to monitor the situation in case of further issues.
Posted 9 years ago. Nov 25, 2016 - 11:26 EST
Monitoring
The issues with our primary queue server appear to be fixed and we've reenabled it for check-in processing. We are handling all new check-ins as they arrive but are still working off the backlog of check-ins from our backup queue service.
Posted 9 years ago. Nov 25, 2016 - 11:14 EST
Identified
Our primary queue server is currently having CPU issues causing check-ins to take longer to both process and respond to clients. We're working with our service provider to address the issue. We've failed over check-in processing to a secondary service but are having issues keeping up with message rates. We expect check-ins processing to be delayed until our main queue server is back at full capacity.