2024-09-30 Elevated error rates on landed cost

What products were affected and what was the impact?

• Landed Cost API

Impact:

• DEGRADED SERVICE

What timeframe did this issue occur?

How was the issue detected?

Synthetic test failures alerted our team.

What functionality was affected?

Increased latency on landed cost quotes, plus a period where landed cost quotes failed.

What problems did this cause?

Landed cost quotes were slow to return and/or failed to return.

What was the resolution of the problem and steps that are being taken for continued follow-up?

The problem was caused by a very large message added to our message queue that could not be consumed due to insufficient resources. The immediate resolution was to increase maximum allocated memory to allow the message queue to clear.

What mitigation solutions will we put in place to prevent this issue from occurring in the future?

To prevent this from happening again, we have decreased the total allowed message size by both the producers and consumers. We are also evaluating techniques to make the message queue more robust to this type of failure.