Facebook temporarily lost data.

Last Sunday Facebook reported a data loss. We are talking about approximately 15% of users’ photos. Loosing your client’s data is the worst thing that could happen to you and reminded me what a guy said once in a tech talk: “The main rules in running an online community service are: Never lose data and never go to jail.”

Facebook has not yet made public the details of what happened but only assured users that their photos will be restored using a backup. The official report states that we are talking about a hardware failure at storage level.

First of some key facts about Facebook

Based on the above numbers it means that they lost approximately 1.5 billion of pictures. Waw!

How is Facebook handling user’s images? Last year Jason Sobel, Manager of the Facebook Infrastructure Group, presented some insights about the current Facebook storage solution and the future one. We don’t know right now whether the new storage solution failed or the old one is to blame.

Writing files using the old way

They were using upload servers and stored images via NFS into a NetApp storage (last year they were planning to replace it). Each image is stored 4 times. This solution experienced heavy workload when processing metadata.

Reading files using the old way

Here all resumes to speed.

The main concerns with the above architecture are:  Netapp storage is overwhelmed, they rely too much on CDNs.

Obviously when your app grows like hell, you start to think that is better to make your own toys, fully customized and optimized for your particular problem. So did Amazon back in 2001 and  Google too.  This is how the Facebook storage was born: Haystack


The answer was to develop in house a distributed file system like GFS (Google File System). Haystack should run on inexpensive commodity hardware, and it should deliver high aggregate performance to a large number of clients.

Haystack is file based and stores arbitrary data in files. For 1Gb disk data file they create 1M in memory index. In this way they have one disk seek which is much better than NetApp which had 3.

The Haystack format is rather simple and efficient, Version number, Magic number (supplied by the client to prevent brute force attack), length, data, checksum.  The index simply stores the Version, Photo key, Photo size, start, length.

Using a Haystack server

To write uses POST
- writes data on disk haystack
- writes data on in memory index
To read uses GET
- uses the in memory index to retrieve the offset
- reads data from the on-disk file

This simple approach allows Facebook to easily balance the reads and writes using Haystack clusters but to speed up the reads they still plan to use CDNs in areas where they don’t have datacenters and Cachr for profiles. This is their first step to create their own CDN network.

Additional readings

Needle in a haystack: efficient storage of billions of photos

Engineering[at]Facebook’s Notes