Network Performance Issues
Incident Report for Finalsite
Postmortem

At approximately 1:25 PM EDT on Friday, September 10th, our team began observing elevated error rates equivalent to 4% of our inbound traffic related to an internal hostname configuration issue. Upon observing this, the team immediately began troubleshooting, engaging Cloudflare engineers to mitigate the impact of this issue. 

During this time, live clients websites were not affected and the scope of impact for this issue was limited to traffic directed at the internal Finalsite hostnames (formatted as *.finalsite.com) and a small subset of features that leverage the internal hostname to communicate. These affected features included file management within the Messages and eNotify modules, and some assets in the Mobile App not loading properly.

By 4:23 PM EDT, mitigation work had been completed, functionality was restored, and we shifted to a monitoring status. At 5:21PM EDT, the issue was marked resolved. 

We apologize for any inconvenience this may have caused, and do not anticipate a recurrence of this issue. We remain committed to our mission in the service of educators around the world and thank you for your continued partnership with Finalsite.

Posted Sep 13, 2021 - 15:45 EDT

Resolved
This incident has been fully resolved. Our sincere apologies for any inconvenience this has caused. Our team is continuing to review this incident and will publish a full post-mortem as soon as our investigation is complete.
Posted Sep 10, 2021 - 17:21 EDT
Monitoring
Our team has put a fix in place for this issue and is continuing to monitor sites closely. Errors related to internal hostnames (*.finalsite.com) should be resolved shortly (if they are not already). We apologize for the inconvenience.
Posted Sep 10, 2021 - 16:34 EDT
Update
We are seeing errors we believe to be related to internal hostnames only (*.finalsite.com). Client hostnames do not appear to be affected. We are continuing to investigate the issue.
Posted Sep 10, 2021 - 14:26 EDT
Investigating
We are currently investigating issues with network performance. We are working to analyze and mitigate this problem. More updates to follow shortly.
Posted Sep 10, 2021 - 13:36 EDT