OVH fire: details of damage and possible cause published
Less than a week after the fire that hit the Strasbourg datacenter, OVH discloses all the data on the damage suffered and the probable cause of the fire. The company has published a detailed report specifying for each service the current status, whether recoverable or lost. As already anticipated by Octave Klaba, the servers and data of the SGB2 building, which suffered the most damage, are for the most part irrecoverable.
OVH fire: the extent of the damage
The fire at the OVH datacenter was as bad as expected, if not more so: the details of the damage and data lost due to the accident leave no room for doubt. OVH has published a report with all the information on the status of the services, specifying the status of the services and physical servers. Unfortunately , almost all of the services hosted in the SGB2 infrastructure have been classified as non-recoverable , and the remainder is still under investigation.
The services of Plesk , the platform for managing the hosting of the various websites, are totally out of order. Hosted in the SGB2 datacenter, their internal backup is also in a non-recoverable state . It goes much better for bare metal services, almost all of them in a recoverable state. The products of this infrastructure are central to OVH, and fortunately they were housed in the other 3 structures, so they did not suffer irreparable damage.
For Private Cloud services, on the other hand, the situation is worse, with more than 50% of the infrastructure not recoverable, even with internal backup. Public cloud services, on the other hand, are for the most part still under investigation, so the extent of the damage is not yet known in detail. In general, all the services have not yet become available again, and recovery operations are still in progress for the recoverable ones . Only the customers who had activated the FTP Backup service were saved: the service was carried out in the Roubaix datacenter.
On the official OVH website you can check the status of the company's services . Many services are still offline and it is not yet possible to access the state of the SGB3 building. The company currently plans to reactivate the SGB1 and SGB4 datacenters this week , along with the network room. For the SGB3 the times are getting longer: the expected timing is 2 weeks from now.
The possible cause behind the fire
According to what reported by Octave Klaba the "culprit" would be a UPS, or an uninterruptible power supply, located in the SGB2 building . In reality, the cause of the fire would seem almost certain, also considering that the offending UPS had been serviced the day before. Furthermore, from the thermal cameras of the firefighters, UPS7 (the one under maintenance) and UPS8 were on fire when help arrived. The cause could be related to the technical intervention, which seemed to have ended correctly.
Investigations are still ongoing and confirmation on the origin of the fire could arrive in the coming days. OVH has deployed its forces to recover what can be saved, giving itself specific priorities: first of all, restoring the services of SGB1, SGB3 and SGB4; some should be available again today. Subsequently, replacement infrastructure will need to be provided to customers using the Roubaix and Gravelines datacenters. Finally, we will take care of providing assistance to those who have not activated the Disaster Recovery Plan.
Customer servers and orders that were in progress when the fire broke out will be fully refunded. OVH asks customers to pay more attention to emails or telephone contacts they may receive during this period: it is possible that someone takes advantage of the incident to carry out phishing or spam attacks . The SGB2 datacenter will be built with more modern and less accident-prone technologies: the structure was built in 2011 and had never been renovated over the years, unlike SGB3.
The article Fire OVH: details of the damage published and the possible cause comes from Tech CuE | Close-up Engineering .