Power Outage
Incident Report for Holy Spirit School
Postmortem

Going forward advisement

  • Core and MDF switches on UPS – Verify that they are on the battery backup device that is located in the rack.
  • Consider UPSs for the three edge switches -  This will keep the switches on for a short period of time as well as all downstream Wifi APs that are powered via POE.  A UPS will not mitigate long term outage events though.
  • Investigate why we had a Duplicate IP event –During a reboot, DHCP will expire all addresses and reissue leases to all devices booting up.  It is possible during a short outage, a duplicate IP event can happen.  In rare occasions a device’s circuitry would have stored power for a short period due to capacitance, thus holding on to its assigned IP.   For this morning’s outage, we should not have had any dupe IPs since we were without power for 4 hours.
  • VOIP - If you hear that a phone isn’t working, they simply need to unplug it and plug it back in.
Posted Mar 16, 2024 - 09:11 PDT

Resolved
Power restored. All systems back online.
Posted Mar 13, 2024 - 04:04 PDT
Update
We are continuing to investigate this issue.

Found that the following WiFI APs did not come back online cleanly.
2nd grade
6th grade
Gym#2
Gym#3
This was due to duplicate IP address errors between the 4 WiFi devices. I remotely rebooted all four of the devices so that they get issued their new IP leases. This resolved the situation.
Posted Mar 13, 2024 - 04:00 PDT
Investigating
School lost power. All online IT systems offline.
Posted Mar 13, 2024 - 00:31 PDT
This incident affected: Campus Electrical Feed, Campus Network, and VOIP Phones.