Power has been fully restored and our systems are coming back online as our technical team checks out systems and brings everything back online. Will likely still be a bit for everything to stabilize but well update this incident to "resolved" once we are comfortable with everything on our side, plus full assurances from the data center provider that power issues are fully resolved and stable.
I apologize again for the outage today, and please contact our support team if you are still suffering from any issues with your VPS services in our US West data center location.
Posted Feb 21, 2018 - 14:43 CST
Monitoring
So further details are now available. This is actually a power issue, PG&E the power company in the area caused a power outage doing work in the data of the data center. Unfortunately the automatic transfer switch to failover to generator power did not operate as designed causing data center wide power issues once the UPS systems were depleted. They have engineers working on repairing the power transfer switch, as well as PG&E (Pacific Gas and Electric) are working on the overall power outage in the area of the data center itself.
We have updated the outage title to reflect the cause of the outage. Unfortunately from remote monitoring it is sometimes hard to tell the difference between network and power issues as a power issue will also cause network issues...
We will update again as we find out better ETAs on recovery efforts all around.
Posted Feb 21, 2018 - 14:08 CST
Update
While we would always like a bit more details the most recent update from our colocation provider for our US - West location just updated us with:
"We have identified the issue that has affected our Fremont 1 datacenter, and are working towards a resolution. Our global network is unaffected. Thank you for your patience."
We are awaiting further details and an ETA from them hopefully very shortly.
Posted Feb 21, 2018 - 13:31 CST
Update
Our data center providers engineering team is still working on the issues they are suffering in the Fremont, CA data center location. Just want to keep everyone updated that we still working and monitoring the situation.
Posted Feb 21, 2018 - 13:11 CST
Identified
Our data center provider has confirmed they are working on wide scale issue in their Fremont, CA colocation data center. We awaiting a final determination of the cause and an ETA to a fix.
Posted Feb 21, 2018 - 12:45 CST
Investigating
We are seeing issues with traffic flowing to and from our US West data center and are investigating with our data center provider. Will update as soon as we know more.