Normal service of the SMX Filtering Gateway has now been restored. We have identified 43 customer configuration changes made between the most recent backup and the outage that we will manually restore over the course of the day.
Posted Dec 19, 2022 - 12:37 NZDT
Update
Access to the SMX email admin portal has been restored.
Posted Dec 19, 2022 - 12:15 NZDT
Update
We have completed the database restore on 1 of 6 nodes, and mail flow through SMX Filtering is now working again, albeit with degraded performance. We are now working to configure replication from the database primary host to the remaining 5 read-only replicas, which will restore the service to full working capacity. There will be further work required to manually recover system changes made between the most recent backup (6AM this morning) and the time at which the database became corrupted.
Posted Dec 19, 2022 - 12:13 NZDT
Identified
Database restore and re-index is in progress, after which we will perform basic health checking before bringing it back into serivce. We will provide a further update in 30 minutes or when service is restored, which ever comes first.
Posted Dec 19, 2022 - 11:52 NZDT
Update
We believe the problem relates to corruption of an internal configuration database. We are working to restore the database from a recent backup now. We have manually disabled the SMX Email Admin Portal to prevent writes to the database while it is being restored.
Posted Dec 19, 2022 - 11:26 NZDT
Investigating
We are aware of problems affecting the SMX NZ/AU Filtering gateway. SMX technical staff are investigating and will provide a further update in 30 minutes or as new information becomes available.
Posted Dec 19, 2022 - 11:22 NZDT
This incident affected: Mail Filtering and SMX Portal.