We believe all the associated issues with the certificate expiration has been resolved. If you need further assistance, please reach out to support. Sorry again for the inconvenience.
Posted Jul 10, 2022 - 10:06 PDT
Monitoring
We have deployed new certificates. All issues should be resolved now, although it may take a while for the agents to notice they are able to reattempt the authentication process. This process can be sped up (or in some case necessary if the agent had given up) by restarting the Rails app, which resets the agent. Please reach out to support if you need additional assistance.
As part of the response to the Heroku security incident earlier this year (https://status.heroku.com/incidents/2413), a token that was used to upload certificates was invalidated without being replaced. Since the renewal script runs rarely the problem was not noticed until today. We had previously set up secondary monitoring to alert us when a certificate is nearing its expiration date (i.e. the automation did not work as expected), but it appears that secondary monitoring had also failed for a different reason.
We are very sorry for the inconvenience.
Posted Jul 10, 2022 - 10:00 PDT
Identified
Our automation have failed to renew/replace the SSL certificates before they expired. We are currently replacing the certificates manually. In the meantime, both the Skylight dashboard and the data collecting servers (used by the agents to submit traces) are inaccessible. We are sorry for the inconvenience.
Posted Jul 10, 2022 - 09:26 PDT
This incident affected: Website (Application) and Data Processing (Application).