p3.zswitch Down (Provisioner)

Incident Report for 2600hz

Postmortem

Resolved - The restoration has been completed and we are moved back to our main Provisioning server.

Prevention Plan:

Near Term: Infrastructure will establish a sync between zones in case this happens again so that we have a failover.
Mid Term: Infrastructure will set up NFS to share the files across the main and backup servers.

Posted Jun 28, 2023 - 08:45 PDT

Resolved

The restoration has been completed and we are moved back to our main Provisioning server.
Prevention Plan:
1. Near Term: Infrastructure will establish a sync between zones in case this happens again so that we have a failover.
2. Mid Term: Infrastructure will set up NFS to share the files across the main and backup servers.

Posted Jun 23, 2023 - 14:00 PDT

Identified

Provisioner switched to the backup server while main server undergoes maintenance due to a hardware fault, no files have been compromised.

Backup server to take an estimate time of 10 hours while the latest information is restored and synchronized

Existing devices will return 404 "not found" from Provisioner until this process is completed.

p3.zswitch.net IP has been switched to 161.38.209.149 (backup server)
This has the implication that firewalls needs to accept this additional IP for HTTP and HTTPS outgoing traffic (Ports 80 and 443).

Clients can add, update and remove devices, however they may need to take the additional step of unlocking them to get them to work, as we have locked everything as a preventive measure.

This should not disrupt calls or other services in Kazoo.

Posted Jun 23, 2023 - 08:11 PDT

Investigating

We have encountered issues with our provisioner server and are currently working to migrate over to a new host. We will keep this page updated with any further updates.

You may have issues provisioning new phones for the next few hours.

Posted Jun 23, 2023 - 04:52 PDT

This incident affected: Advanced Provisioner.