We have not seen any further examples of the issue overnight so are confident that the issue has now been resolved. We apologise sincerely for the disruption that this may have caused.
Posted Sep 18, 2021 - 06:56 UTC
Monitoring
Dear Customer,
We are pleased to advise you that we are confident that this issue has been resolved and that you can now access the solution as expected.
We would like to take this opportunity on behalf of Fourth to apologise profusely for any inconvenience that this has caused.
Fourth takes issues such as these seriously and we can assure you that the root cause will be reviewed with the utmost priority in order to prevent recurrence.
We will continue to monitor performance for the foreseeable future.
Once more, please accept our apologies and for the inconvenience this has caused you and your organisation.
We'd also like to take this opportunity to thank you for your patience and understanding.
Posted Sep 17, 2021 - 17:43 UTC
Update
Dear Customer,
As you are aware we are currently experiencing degraded performance issues for some customers when accessing parts of the system.
As soon as the issue was reported we quickly traced and identified issues within the network/connectivity space.
As you may be aware from recent updates, we are advising a potential workaround for some customers as follows:
1. Use a device connected to a different network as opposed to the network you are experiencing the issue on.
- If the issue is affecting a mobile device, try a web browser on a laptop.
- If a laptop is affected try a mobile device on a 4G network.
We do acknowledge that this may not work for all customers.Our team is currently working with our network and ISP providers to identify the issue and restore normal performance levels.
We are sincerely sorry for the inconvenience this is causing and will provide further updates as soon as possible.
Thank you for continued patience.
Posted Sep 17, 2021 - 14:03 UTC
Update
Our teams continue to work to solve the issue, whilst regretfully we do not as yet have a final solution, we can provide the following update in a bid to outline the complex nature of the issue.
The complexities The issue is affecting a small subset of customers, other customers using the same systems are not seeing the issue. We cannot reproduce the issue in our test systems - consequently, we have engaged with some affected customers who have allowed us access to help diagnosis. The impacted customers do not have any other reported connectivity issues. Those affected customers do not share the same ISP, although there is a small amount of location and ISP clustering.
What we have done We have identified connectivity issues for some locations when connecting to certain parts of our platform We have cycled every layer of our infrastructure to clear any potentials issues We have investigated DNS and network routing Extensive tests have been carried out with some affected customers to fully identify and scope the issue; this has enabled us to rule out certain parts of the system
What we have found We are able to reproduce the issue across servers and platforms within impacted customer systems and can reproduce the issue across production and non-production environments within said systems.
The distinguishing factor between those systems where we can reproduce the issue and those where we cannot appear to be related to the public IP address range they reside on. Those systems that reside on subnet A are impacted, those on subnet B for example are not.
We are currently investigating DDos, Data Centre and Internet routing for possible causes.
It goes without saying this is our highest priority with every effort is being made to resolve the issue as quickly as possible.
We apologize for any inconvenience and are ensuring every effort is being made to restore normal system performance.
Posted Sep 16, 2021 - 18:56 UTC
Update
As you are aware we are currently experiencing degraded performance issues for some customers when accessing parts of the system.
As soon as the issue was reported we quickly traced and identified issues within the network/connectivity space.
There is a potential workaround for some customers which is to use a device connected to a different network. So if the issue is affecting a mobile device, try a web browser on a laptop, or if a laptop is affected try a mobile device on a 4G network. We do acknowledge that this may not work for all customers.
Our team is currently working with our network and ISP providers to identify the issue and restore normal performance levels.
We are sincerely sorry for the inconvenience this is causing.
Posted Sep 16, 2021 - 12:06 UTC
Update
We are continuing to investigate this issue and are working to identify the affected IP address ranges.
There is a potential work work around for some customers which is to use a device connected to a different network. So if the issue is affecting a mobile device, try a web browser on a laptop, or if a laptop if affected try a mobile device on a 4G network. We do acknowledge that this may not work for all customers.
We apologise for any inconvenience that this is causing.
Posted Sep 15, 2021 - 12:25 UTC
Investigating
We have reports of a number of customers intermittently having issues logging in to Engage. This is affecting a small subset of customers and appears to be geography / ISP related.
We are working to identify and resolve the issue and will provide a further update in an hour.