Internet and Cloud Outage - New Zealand
Affected
- Resolved · UTC
The incident is now resolved and all services are back online.
Post Mortem:
At 6:20am, our network operations were notified that there was an issue at our core site 220Q. Cloud services had failed and the ISP network was failing to authenticate users.
An immediate response was launched. Due to failure to authenticate, all users timed out and disconnected shortly after.
At 6:25am, a technician was dispatched to 220Q to investigate. At 6:35am the technician arrived and ascertained that a PDU (power distribution unit) had failed, powering multiple levels of networking equipment.
At 6:40am, the devices were swapped to an alternate PDU. All devices started to come back online, and by 6:45am, the entire network was considered restored, excluding one VM server which required manual intervention. This was resolved shortly before 7am.
We are commencing our PDU replacement program during Christmas to ensure this does not happen again. There will be minor downtime while this occurs, but you will be notified of this well before it happens.
This case is now closed. The outage lasted 40 minutes. If you have any further questions or queries, please phone 0508-100-200. - Investigating · UTC
We are currently investigating this incident. We are aware that some internet connections and cloud services within Auckland are currently inaccessible. A further update will be provided in 1 hour.