The incident is now resolved and all services are back online.
Post Mortem:
At 6:20am, our network operations were notified that there was an issue at our core site 220Q. Cloud services had failed and the ISP network was failing to authenticate users.
An immediate response was launched. Due to failure to authenticate, all users timed out and disconnected shortly after.
At 6:25am, a technician was dispatched to 220Q to investigate. At 6:35am the technician arrived and ascertained that a PDU (power distribution unit) had failed, powering multiple levels of networking equipment.
At 6:40am, the devices were swapped to an alternate PDU. All devices started to come back online, and by 6:45am, the entire network was considered restored, excluding one VM server which required manual intervention. This was resolved shortly before 7am.
We are commencing our PDU replacement program during Christmas to ensure this does not happen again. There will be minor downtime while this occurs, but you will be notified of this well before it happens.
This case is now closed. The outage lasted 40 minutes. If you have any further questions or queries, please phone 0508-100-200.
