Cellular devices not able to get online
Incident Report for Particle
Resolved
The cause of the cellular outage has been addressed. Most devices have already reconnected, and the rest will continue to come online as they're able to. If they're not yet connected, don't worry. They'll continue to try until network conditions allow it.

If after 4pm Pacific if your devices still haven't recovered, please reach out to us and we'll look into the issue further. You can reach our support staff here: https://docs.particle.io/support/support-and-fulfillment/menu-base/

What caused the outage?

One of our telephony carriers in the US performed some network maintenance that was supposed to be zero downtime and have no noticeable effect for us or you. They didn't notify us ahead of time. The maintenance work failed and they had to roll the change back. The rollback affected many Electrons with Particle SIMs during the last 18 hours.

During the outage, an Electron trying to get online could establish a cellular connection, but wouldn't be able to establish a data connection. Devices with existing data connections would have stayed connected. Unfortunately, some devices got knocked offline during the rollback and then couldn't re-connect.

Our carrier reported they'd fixed the issue around 7:40am PST and devices started to connect again gradually. We also saw this on our end.

What are we going to improve on?

While an outage like this was out of our immediate control, it took us longer than it should have to identify the scope of the issue and be able to communicate about the outage to you and our telephony partners to enable them to address the issue more quickly. We've got some good tools for helping understand device-level, and product-level connectivity, but our tools for monitoring and alerting on carrier-level issues need to be improved. We'll be working on improving those in the coming days.

Thanks for your patience and support!
Posted Nov 07, 2017 - 14:44 PST
Monitoring
We've identified the issue and are monitoring its resolution. We're seeing much higher connection rates, and error rates have fallen. It seems most devices that had trouble connecting are now able to connect.
Posted Nov 07, 2017 - 09:54 PST
Investigating
Some Electron cellular devices are not able to reconnect to the Particle cloud after our telephony partner maintenance window on Monday. The devices blink green indefinitely. We are investigating the root cause.
Posted Nov 07, 2017 - 06:27 PST