Granicus Issue Notification: govService LIM Issues - Integrations Failures
Incident
Report for Granicus System
Resolved
Granicus Engineering teams have identified and resolved the issue impacting LIM / Integrations. We apologize for any inconvenience this service issue may have caused and thank you for your patience.
Product: govService
On Thursday June 3, 2021 at 8:45 am BST it was reported that the Local Integration Module (LIM) integrations were failing for a subset of our customers.
The team initially suspected the issue was caused by the software release the prior evening and decided to rollback the release. The rollback did not resolve the issue.
Root Cause
The issue was caused by the Proxy server timing out when it did not receive the full response from the LIM.
Action to Resolve
A new proxy server was built and impacted accounts were moved over to the new Proxy in a controlled manner.
If you have any further questions or would like more information about this resolved issue, please contact us at Support.Granicus.com.
Thank you,
Granicus Customer Support
Product: govService
On Thursday June 3, 2021 at 8:45 am BST it was reported that the Local Integration Module (LIM) integrations were failing for a subset of our customers.
The team initially suspected the issue was caused by the software release the prior evening and decided to rollback the release. The rollback did not resolve the issue.
Root Cause
The issue was caused by the Proxy server timing out when it did not receive the full response from the LIM.
Action to Resolve
A new proxy server was built and impacted accounts were moved over to the new Proxy in a controlled manner.
If you have any further questions or would like more information about this resolved issue, please contact us at Support.Granicus.com.
Thank you,
Granicus Customer Support
Monitoring
All customers should now be back to normal regarding delayed / failed integrations although a little slower than usual for the moment.
Signup and Password Reset Emails are all now working and the backlog queue is processing through. Customers using ‘ Save Emails’ may still experience some issues for the time being.
Failed integrations can be rerun as normal via the Integrations Manager.
Product: govService Integrations
Thank you,
Granicus Customer Support
Signup and Password Reset Emails are all now working and the backlog queue is processing through. Customers using ‘ Save Emails’ may still experience some issues for the time being.
Failed integrations can be rerun as normal via the Integrations Manager.
Product: govService Integrations
Thank you,
Granicus Customer Support
Update
We are continuing to resolve the integration issues. As part of this work, we have noticed some sites do not have both of our reverse proxy IPs whitelisted for communications with your LIMs. This whitelisting is normally a standard part of initial platform implementation prerequisites.
For any sites still experiencing issues, please can you check with your IT team to ensure that both: 79.125.23.126 and 54.75.228.160 are whitelisted for communication to your LIM as a matter of urgency as we are unable to move your site over to the new configuration otherwise.
Please let us know via your support ticket if you make any whitelisting changes and we can then transfer you over.
Product: govService Integrations
Thank you,
Granicus Customer Support
For any sites still experiencing issues, please can you check with your IT team to ensure that both: 79.125.23.126 and 54.75.228.160 are whitelisted for communication to your LIM as a matter of urgency as we are unable to move your site over to the new configuration otherwise.
Please let us know via your support ticket if you make any whitelisting changes and we can then transfer you over.
Product: govService Integrations
Thank you,
Granicus Customer Support
Identified
Granicus Engineering teams have identified a solution to the govService Integrations issue and are progressively moving sites over to the new configuration in a controlled manner. Please bear with us whilst we move remaining affected sites over throughout today. We will update customers via their individual tickets to let them know when they have been moved over. Our teams will continue to monitor this until we have the issue fully resolved.
Product: govService Integrations
Thank you for your patience
Granicus Customer Support
Product: govService Integrations
Thank you for your patience
Granicus Customer Support
Update
Granicus Engineering teams have mitigated the earlier issues with delayed / failing integrations. Integrations are now running. Customers may experience some platform slowness around submissions. Email integrations have been temporarily halted as we have identified some email routing issues which require further investigation to establish a resolution before we re-enable. This means Signup and Password Reset Emails will not process. Our teams are continuing to work on this issue overnight and further updates will be provided tomorrow.
Product: govService Integrations
Thank you,
Granicus Customer Support
Product: govService Integrations
Thank you,
Granicus Customer Support
Update
Granicus Engineering teams continue to investigate the govService LIM issue. We are aware that Email integrations in particular are experiencing issues. Our teams will provide further updates until we have the issue fully resolved.
Product: govService Integrations
Thank you,
Granicus Customer Support
Product: govService Integrations
Thank you,
Granicus Customer Support
Update
The LIM issue is now only affecting a small subset of customers. We will communicate appropriately via individual tickets and continue to work on the root cause as a priority.
Investigating
Granicus Engineering teams are currently investigating govService Integration Failures. Our teams are working on this issue as priority and further updates will be provided as soon as possible.
Product: govService Integrations (LIM)
New integrations running from now should be back to normal. There is a queue backlog for earlier integrations which may take time to run through.
As part of this issue, we had to roll back last nights release (Release 477) and this will be re-applied as soon as possible, once the root cause of this LIM issue has been established.
We apologise for any inconvenience caused and appreciate your patience whilst we fully resolve this issue.
Updates on this issue will continue to be provided on status.granicus.com, for other issues or questions please contact us at support@granicus.com.
Thank you,
Granicus Customer Support
Product: govService Integrations (LIM)
New integrations running from now should be back to normal. There is a queue backlog for earlier integrations which may take time to run through.
As part of this issue, we had to roll back last nights release (Release 477) and this will be re-applied as soon as possible, once the root cause of this LIM issue has been established.
We apologise for any inconvenience caused and appreciate your patience whilst we fully resolve this issue.
Updates on this issue will continue to be provided on status.granicus.com, for other issues or questions please contact us at support@granicus.com.
Thank you,
Granicus Customer Support
This incident affected: govService (govService U.K.).