Concepts
Warm storage and erasure coding
Intuition
Imagine a records archive with old documents that almost nobody consults. Keeping three complete copies of each paper would be a waste of shelf space; it is cheaper to split the content into pieces and keep a few extra spare pieces, so that if some are lost, the original can be rebuilt from the ones that remain. In exchange, recovering a document after a loss takes more work than simply grabbing a ready-made copy.
How it works
In Facebook's BLOBs, the read rate drops with the age of the data, and most objects of the analyzed types end up warm: few reads per terabyte, so the bottleneck becomes capacity rather than throughput (analysis of the f4 paper). Haystack, which serves the hot content, keeps three replicas with RAID-6 on each machine, an effective replication factor of 3.6. f4 replaces whole copies with Reed-Solomon (10,4) erasure coding: each stripe has 10 data blocks and 4 parity blocks, spread across different racks, and any 4 lost blocks can be reconstructed, at a cost of 1.4 times the size of the data. This is simple to operate because f4 only stores locked volumes, which accept only reads and deletes, with no appends or overwrites that would force parity to be recomputed; deletion destroys the BLOB's encryption key instead of rewriting the stripe. In a normal read, the storage node returns the data directly; if the data block holding the BLOB is unavailable, backoff nodes reconstruct on demand only the requested BLOB, from the other blocks of the stripe, while rebuilder nodes rebuild whole blocks in the background, rate-limited so they don't hurt users. To survive the loss of a datacenter, the first solution keeps a second replica of the cell in another datacenter (1.4 × 2 = 2.8); the next one pairs volumes from two regions and writes their XOR to a third, so that the loss of any one of the three can be recovered from the other two, which gives (1.4·2 + 1.4)/2 = 2.1. The price is lower maximum read throughput and a higher cost of reading and reconstruction when something fails; in the normal case, reads are slightly slower than Haystack's.
In practice
The f4 paper reports that, for reads within the same region, f4's median latency is 17 ms versus 14 ms for Haystack, which the authors consider sufficient for a good user experience, and that f4 complements Haystack rather than replacing it. The choice makes sense when the data is immutable, rarely read per terabyte, and capacity-bound, and the cutoff between hot and warm comes from analyzing the access pattern by age (in the paper's analysis, about one month for most types and three months for Photos). For data that is still hot or heavily read, the fully replicated tier remains the right choice, since it offers more read throughput and does not depend on reconstruction when a copy fails.
Use it in a design
Practice the cases that rely on this concept in the System Design school.