Elevated error rates in recommendation services and API
Incident Report for Reflektion, Inc
Resolved
All systems are functional . We will continue to monitor from our end.
Please continue to reach out to customer support if you see any issues.
Posted Dec 07, 2021 - 23:06 PST
Monitoring
The error rates are back to normal levels and we have been able to verify our systems are up and running. We are continuing to monitor from our end.
Posted Dec 07, 2021 - 20:45 PST
Update
Applications are back closer to 90% and We'll keep you posted once it is back to 100%.
Posted Dec 07, 2021 - 17:05 PST
Update
Many services have already recovered, however AWS teams are working towards full recovery across services.
Posted Dec 07, 2021 - 16:41 PST
Update
We can confirm Reflektion Customer Engagement Console (CEC) is back to normal.
Posted Dec 07, 2021 - 16:33 PST
Update
AWS US east coast region (us-east-1) is experiencing outage which affects world’s largest sites, and we are also affected by this outage.

We are seeing Reflektion services being gradually restored as AWS continues to make progress on issue resolution.


We also have been working for last few hours to create alternative end points for our services.

These alternative end points will be disabled once AWS resolves their issues. We will inform you once AWS resolved the issue so you can revert back the change you made. These end points will no longer work after issue resolution.


If you continue to experience degraded services for API integrated widgets and modules, please contact support@reflektion.com for more details regarding temporary endpoints.

For JS integrated sites and modules, we observed Gateway timeout for subdomain based Beacon endpoint, you can adjust the beacon endpoint to be the non-subdomain version. Please reach out to support@reflektion.com for more details.
Posted Dec 07, 2021 - 13:12 PST
Update
Reflektion Customer Engagement Console (CEC) is also down at the moment.

We are continuing to work on a fix for this issue.
Posted Dec 07, 2021 - 11:27 PST
Update
We are continuing to work on a fix for this issue.
Posted Dec 07, 2021 - 11:07 PST
Update
Our customer support portal (zendesk) is also down at the moment. To contact support , please email support@reflektion.com
Posted Dec 07, 2021 - 10:13 PST
Identified
We have identified this issue is due to an ongoing AWS outage. We are keeping this incident open while we wait for AWS systems to recover.
Posted Dec 07, 2021 - 09:46 PST
Update
We are continuing to investigate this issue.
Posted Dec 07, 2021 - 09:14 PST
Update
Promotional email services are also affected at this point of time. We are continuing to investigate this issue
Posted Dec 07, 2021 - 09:13 PST
Update
We are continuing to investigate this issue.
Posted Dec 07, 2021 - 08:52 PST
Investigating
We are seeing an increase in the error rates (1% - 2%) across recommendation and API services. We are investigating the issue. Will post updates as and when we see improvements.
Posted Dec 07, 2021 - 08:33 PST
This incident affected: Production Recommendations, Production API, Production Customer Engagement Console, and Production Email Services.