The OIDP service is a load balanced OpenLDAP implementation that serves as an Oracle Internet Directory service. Oracle clients can be pointed at this service to resolve the connection information for Databases. This simplifies the migration of databases as clients do not need to be individually updated when the database location changes.
On Dec 7th at 05:54:50 AM the OIDP service became unavailable. Customers who were using this service with TOAD or other Oracle clients were no longer able to connect to Databases. As a workaround, users could manually specify each database connection if known, however this requires changing settings and files so most users chose to wait until the service was restored at 12:52 PM.
This service should be available to customers 24/7 except during pre-defined maintenance windows.
On Dec 7th at 05:54:50 AM a network disruption occurred which caused the default node health monitor in the loadbalancer to be disconnected and enter a failed state. This caused the nodes to be marked as offline and so the load balancer quit routing traffic to the servers. When the network disruption was resolved the node monitor did not reset automatically causing the nodes to continue to be marked as offline despite passing other health checks. May be due to this known issue.
Monitoring triggered a PagerDuty incident at 6:07 AM that the service was down, but it was marked as low-urgency and so didn’t escalate until 8:00 AM. Technicians began troubleshooting at around 10:14 AM, but the urgency was not properly understood initially and so the focus was on identifying why the load balancer did not mark the node as healthy again instead of restoring service. At 12:52 PM after calls to the service desk were relayed about TOAD users being impacted, the issue was resolved by manually disabling the nodes’ health monitor and re-enabling it. This caused the nodes to once again be marked as online and service restored.
Despite monitoring correctly identifying and alerting to the issue, response time was low due to lack of technicians comfortable with the F5 and a misunderstanding of the impact. Team members will do cross training and make sure everyone understands how to address this issue in the future and to respond in a more timely manner.
December 8, 2022 - Discuss the urgency with team members and cross train on how to resolve the issue.