The Situation
On a Monday morning in July of 2015, the phone rang at the Tech North office and a long-time client said those dreaded words ‘We’ve had a flood’! So began the process of getting this business back on its feet again. The clock was ticking – payroll was due.
Luckily, this was one of our totalCare backup clients. We had a Datto Siris device on-site so we should have a solid backup in place. These devices are great, proving a hot-spare server that can be spun up in a virtual environment, in the event of a catastrophic server failure. We have a continuous backup of the server on-site and a nightly cloud based backup located in a remote data centre.
Site Evaluation
A problem became immediately apparent when we toured the facility after the flood. The flood occurred because of a failed plumbing fitting in an upstairs bathroom. The water had flowed across the flooring to the walls to a depth of 10 cm. It flowed out the door and down the interior walls where it flowed across the drop ceiling into the basement area. One of the channels that the water flowed terminated in the server room. This, in turn, terminated the server – and the backup appliance! Both the server, the Datto Siris backup appliance and all of the networking equipment (switch and router) along with a large UPS had been hit by a continuous stream of water over a period of 24 to 48 hours. The smell of fried circuitry and smoke permeated the room. This wasn’t good.
Equipment Evaluation
After determining that the equipment was safe to work on because the power was off, our crew removed it from the server closet and took it back to our technical work shop. We had to pour several cups of water out of the server chassis, backup appliance and network equipment. The server had failed completely although the hard-drives were likely sound. The backup appliance was in similar shape and the network gear was a complete write-off with the exception of a dedicated Wireless Access Point which had been mounted on a wall.
Luckily, all of the desktop computers were located on the tops of desks on the top floor of the building, so they were not impacted by the flood. Two laptops and an LCD projector were flood damaged and needed to be replaced as well.
Disaster Recovery and Business Continuity
Since the server was nearing end-of-life we agreed that it should be replaced completely. the client didn’t want to trust the older, potentially flood damaged hard disk drives. under normal server failure conditions (think about that for a moment) we would have used the backup appliance to spin up a virtual machine (VM) of the last backup captured Friday night. This was impossible because the backup appliance had failed. We recovered the image of the backup VM from the backup appliance and copied the image to a hard-drive that we could install in a spare server that we keep available for recovery and testing purposes.
We set the server up in spare office space along in an isolated network and transported the bookkeeper’s desktop computer to this space. We brought the server up on the loaner server and connected the key desktop computer and invited the bookkeeper to evaluate the system to ensure she could work on her time-sensitive payroll. All was good. No data loss and we managed to ensure business continuity.
Equipment Replacement
After contacting Datto and informing them of the disaster recovery situation unfolding, they immediately couriered out a replacement Siris device. Since we effectively had the server in a running state, when the new backup device was available it was put into service backing up the server and streaming the data off-site to the remote facility. The critical client data continued to be protected while we began the process of permanently replacing the equipment. Ultimately the client moved the server into a better location and installed a proper rack to contain the network, server, backup appliance and UPS. The equipment is now protected from falling water and elevated from the floor, in case of flood.