Coveralls is in Read Only mode while we work on updating the system. Sorry for the inconvenience.
VIGILANCE! Check this page Any time you notice a problem with coveralls
[Re-Opened] Github status updates failing intermittently
Incident
Report for Coveralls
Resolved
We are closing this incident for now: We have not been able to reproduce the issue with any data from customer incidents, and, in the past 12 hours, we have seen no further failing status update jobs in our background queues where we have been monitoring them. This further supports the notion that status updates were failing (intermittently) due to a factor external to coveralls.io. We have automated some monitoring around this, but if you experience the issue more than once (in a row), please let us know at support@coveralls.io. Please share the relevant Coveralls build URL(s) in your email.
Update
We are keeping this issue open on reports of intermittent (but ongoing) failures to receive Github status updates. We have not identified a root cause but are addressing customer issues individually as we continue to monitor and investigate.
Investigating
Re-opening this incident from yesterday, THU, APR 14, since we are receiving more reports this morning:
We are getting reports of Github status updates not arriving at Github for repos that have been fine in this regard for a long time. We are noticing an uptick in background job failures responsible for these status updates. In most cases, these failures resolve immediately by re-trying them. But some fail a few times before succeeding. We suspect there's an issue with the Github API (though not according to https://www.githubstatus.com/), or an isolated issue between coveralls.io and github.com, that's resulting in behavior like rate limiting. There is no known basis for actual rate limiting, but this is the effect.
Please know that any failing status updates will be retried in exponential back-off fashion (so they should eventually succeed) and that we are monitoring closely and will take any action we can to improve things.
We are getting reports of Github status updates not arriving at Github for repos that have been fine in this regard for a long time. We are noticing an uptick in background job failures responsible for these status updates. In most cases, these failures resolve immediately by re-trying them. But some fail a few times before succeeding. We suspect there's an issue with the Github API (though not according to https://www.githubstatus.com/), or an isolated issue between coveralls.io and github.com, that's resulting in behavior like rate limiting. There is no known basis for actual rate limiting, but this is the effect.
Please know that any failing status updates will be retried in exponential back-off fashion (so they should eventually succeed) and that we are monitoring closely and will take any action we can to improve things.
This incident affected: GitHub.