As of 1411 UTC (0911 EST) on 29 Dec 2020, our third-party domain name registrar has removed the wasabisys.com domain from their blacklist. We have seen traffic reach normal levels for most of our customers. Due to the time it takes DNS to refresh, it may take several hours for all affected customers to fully access the Wasabi service.
To summarize this incident, our domain name registration was suspended due to concerns over publicly accessible malicious content that was uploaded by one of our newly subscribed customers. Our domain name registrar attempted to notify Wasabi of this problem but the email report of this problem from them went to the incorrect Wasabi email address. After the domain name registrar did not receive a response from Wasabi, they blacklisted the wasabisys.com domain. Upon learning of the malicious content report, Wasabi did immediately suspend the associated customer account for terms of service violations. We also contacted the domain name registrar as soon as we identified the problem. However, despite our escalation attempts, it took over 13 hours for them to reactivate the wasabisys.com domain.
We are preparing a more detailed root cause analysis (RCA) to share with affected customers and the following actions are planned for the near future: * Enhancing our existing policies for handling abuse reports from third-party service providers * Replace our current domain name registrar with one more suited for our global cloud service * Reviewing our policies regarding anonymous public access to customer buckets
We thank all of our customers and partners for their patience and support through this service event.
Posted Dec 29, 2020 - 17:57 UTC
Monitoring
UPDATE: Our domain has been reactivated and DNS updates are beginning to propagate. We expect that you should now be able to reach all of the wasabisys.com URLs with success. We’ll continue to monitor the system and provide another update again by noon EST
Posted Dec 29, 2020 - 14:23 UTC
Update
We continue to rely on the restoration of the wasabisys.com domain from our registrar. No ETA has been provided yet, but we’ll continue to update the status page every 2 hours or at any point where new updates are warranted.
Posted Dec 29, 2020 - 12:40 UTC
Update
We are currently awaiting the restoral of the wasabisys.com domain by our third-party domain name provider. We do not have a restoral ETA at this point because we are dependent on the third-party domain name provider to take actions.
Posted Dec 29, 2020 - 04:07 UTC
Update
Our issue has been isolated to a problem with our wasabisys.com domain as a result of some malicious content hosted in one of the regions. The content has been removed, and we are working to resolve the impact of this. We'll continue updating this status as we try to restore normal services.
Posted Dec 29, 2020 - 01:25 UTC
Update
We've narrowed this down to the domain hosting service and should be able to turn this around shortly. We'll continue with another update at about 02:00 UTC.
Posted Dec 29, 2020 - 00:16 UTC
Identified
Just an update that some customers continue to see issues with DNS servers not resolving the Wasabi endpoints such as s3.wasabisys.com or s3.eu-central-1.wasabisys.com, etc correctly. We’ve had customers report issues with Google DNS (8.8.8.8) and Cloudflare (1.1.1.1). OpenDNS (208.67. 222.222) has shown a bit better. We’re working with our authoritative DNS provider to try to further isolate the issue. We’ll continue to keep the status page updated as they report progress.
Posted Dec 28, 2020 - 21:18 UTC
Investigating
We are currently investigating reports of issues resolving some endpoints including s3.eu-central-1.wasabisys.com s3.us-west-1.wasabisys.com and s3.wasabisys.com. We are investigating to try to determine the source of the issue. We will update this page as soon as we have more details.
Posted Dec 28, 2020 - 19:38 UTC
This incident affected: US-Central-1 (Texas), US-East-1 (N. Virginia), US-East-2 (N. Virginia), US-West-1 (Oregon), EU-Central-1 (Amsterdam), AP-Northeast-1 (Tokyo), and Wasabi Management Console.