[Elements] C14 Cold Storage service degradation
Incident Report for Scaleway
Resolved
This incident has been resolved.
Posted Jul 26, 2021 - 16:48 CEST
Update
Thanks to the work of our team, including modifications, as well as production releases, the incident is now resolved.
You should not be impacted anymore and the objects in the "restore" state should be available again soon via the normal delay (up to 72 hours).
Posted Jul 26, 2021 - 16:48 CEST
Update
Restores and delete operations from C14 Cold Storage are slowly being proceeded on AMS and PAR.
Our product team is still working on fixes which will be deployed soon.
Posted Jul 22, 2021 - 17:12 CEST
Update
We still observe latencies from AMS, the team continues to work and monitor the activity.
Posted Jul 19, 2021 - 10:55 CEST
Monitoring
Both issues were fixed. Please note that performance may be degraded during a few hours to catch up.
Please be sure that all requests will be proceeded. Our team is still monitoring the situation.
Posted Jul 09, 2021 - 17:16 CEST
Update
Our product team applied a patch to fix multiple issues, which includes restore from C14 Cold Storage, and upload to C14 Cold Storage.
Upload and restore process are available now. Our team is still monitoring the situation, due to high loads.
Posted Jul 09, 2021 - 12:50 CEST
Update
We are continuing to investigate this issue.
Posted Jul 09, 2021 - 11:03 CEST
Investigating
We've detected a global issue impacting upload (directly to C14 Cold Storage), and restore (from C14 Cold Storage).
Our product team engineers are working on it.
Posted Jul 08, 2021 - 13:58 CEST
Update
Our product team has stabilized the software, the restoration jobs from C14 Cold Storage will be restarted tomorrow.
Posted Jun 30, 2021 - 18:48 CEST
Update
Update on the situation: while the C14 Cold Storage platform is built on custom hardware, we are struggling with some instabilities related to the kernel (both old and new) as well as user land software. Our product team has been able to fix several of the issues that appeared after an enhancement that used the hardware more heavily.

Your data has not been lost nor corrupted, it is safely saved on disk. The metadata has not been corrupted either as it is stored elsewhere. We are working on how to stabilize the software, both kernel and user land. We will post another update on the situation tomorrow, and will write a complete post-mortem as detailed as we can once the issue is resolved.
Posted Jun 29, 2021 - 18:51 CEST
Update
Our product team has identified the issue and will physically intervene at DC4 tomorrow in order to address it. Only data restoration is currently impacted. Note that affected data is not lost, as disks themselves are not facing any kind of issue. More information to come once the intervention has been performed.
Posted Jun 28, 2021 - 19:25 CEST
Update
We are still experiencing a partial unavailability of our C14 Cold Storage service. Part of the service may be unavailable.
Our product team is currently on site.
Posted Jun 18, 2021 - 16:58 CEST
Identified
Our product team has identified the issue and is working on restoring the service.
We are still facing a partial unavailability of C14 Cold Storage on the "fr-par" region.
Posted Jun 14, 2021 - 16:03 CEST
Investigating
We have noticed service degradation on the C14 Cold Storage platform due to hardware issues.
Our product team is investigating in order to find a solution.
Posted Jun 10, 2021 - 09:58 CEST
This incident affected: Elements - Products (C14 Cold Storage).