Hetzner Online GmbH, a German cloud services provider, told some customers this week that their data had been irreversibly lost and were provided a 20€ compensation in online credit.
Hetzner, which operates several data centers in Germany and Finland, suffered a rare occurrence of multiple hardware failures that have wiped some customers’ snapshots, with no way of recovering them.
The news of this data loss comes from a letter shared by a Hetzner customer on Y Combinator stating one of Hetzner’s clusters located in Nuremberg was impacted by a very unfortunate series of events where multiple disks failed in brief succession.
According to that notice, Hetzner’s cloud services achieve data security by copying the snapshots across three hard disks, so even if two fail, the data will still be available.
In this case, though, not even that redundancy level was enough to save the customers’ data.
“Unfortunately, we have to inform you that there was a data loss incident that affects a small amount of your snapshots on Hetzner Cloud. All snapshots you create are stored on our highly available storage systems,” reads the Hetzner email to customers.
“The snapshot contents are distributed over multiple internal servers and data is stored in a way that allows up to two separate disks to fail without impacting data integrity. This means the snapshot can still be accessed, even if two disks fail at the same time.”
“Due to a recent, very unfortunate series of events in one of our clusters, multiple disks failed in short succession and caused a small number of snapshots to become unavailable. We immediately tried to recover the affected snapshots but unfortunately the data is lost and we have exhausted all our options.”
A small compensation
Hetzner attempted to ease customers’ data loss pain by providing a 20€ Cloud Credit to their account but with the stipulation that it must be used within a year.
“While we know that this will not bring back your data, we still hope that you will accept the gesture,” writes the letter.
This amount has also been confirmed by the German tech news outlet Golem.de, which claims to have seen the associated emails of several affected Hetzner Cloud customers.https://beec2e786e7bacccb03c266e3a144565.safeframe.googlesyndication.com/safeframe/1-0-38/html/container.html
According to the same source, who has had a limited exchange with a Hetzner representative, the cloud provider lost a total of 1,500 snapshots. While this may seem like a small number, the loss of data can be devastating to those customers who are affected.
Bleeping Computer has asked Hetzner to verify that number and share more details on the amount of data lost, but we have yet to receive a response.
Cause of data loss is still unknown
Hetzner hasn’t shared many details about what caused the incident, and neither its status page nor its social media channels have any relevant announcements.
Potential reasons for such high-impact data center incidents include hacking of unprotected data center management systems, catastrophic backup errors, physical disasters such as fire, or simply the unfortunate coincidence of multiple hardware failures at the same time.
Last month, Hetzner published a statement explaining that it erroneously blocked a Ukrainian war information site, saying it was a mistake based on misleading technical information, not hiding any political motivation.
Source: https://www.bleepingcomputer.com/news/security/hetzner-lost-customer-data-and-gave-20-as-compensation/
