Papers
The Google File System
Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung · SOSP 2003 (19th ACM Symposium on Operating Systems Principles), 2003
The problem: storage for Google's own workloads, not general-purpose file access
By the early 2000s Google needed to store and process enormous amounts of data — crawled web pages, logs, index shards — across clusters built from ordinary, failure-prone commodity machines, not specialized reliable hardware. The authors argued that existing distributed file systems were designed around assumptions that did not match this reality. They started instead from the actual shape of Google's own workloads: files were routinely huge, multi-gigabyte objects rather than small documents; most files were written once, by appending new data, and afterward only ever read, often sequentially by many machines at once (a batch analytics job scanning a huge dataset, for instance); and because the system would run on thousands of cheap machines, component failure — disks, machines, even network links — had to be treated as a routine, expected event rather than a rare exception requiring heroics. A further consequence of appends being the dominant write pattern was that many clients might want to append to the same file concurrently, and the system needed to make that both fast and correct without expensive coordination. Rather than trying to satisfy a generic POSIX-like file API, the team chose to co-design the file system's API with the applications that would use it, accepting a non-standard interface in exchange for a system that fit its actual workload far better than an off-the-shelf answer could.
The core idea: one master, many chunkservers, large fixed-size chunks
GFS organizes a cluster into a single master process and many chunkservers, accessed by many clients. Files are split into fixed-size chunks — 64 megabytes each, unusually large compared to typical file system blocks — and each chunk is stored as a plain file on a chunkserver's local disk, replicated (by default across three chunkservers) so that losing any one machine does not lose data. The master is deliberately kept out of the data path: it holds only metadata — the directory-like namespace, the mapping from each file to its ordered list of chunks, and the current location of each chunk's replicas — while the actual bytes of a chunk flow directly between clients and chunkservers. A client asking to read or write first talks to the master to learn which chunkservers hold the relevant chunk, caches that answer for a while, and then talks to the chunkservers directly, so the master never becomes a bottleneck for bulk data transfer even though every metadata operation passes through it. Choosing such a large, fixed chunk size was itself a deliberate trade-off: it keeps the master's per-chunk metadata small enough to fit in memory even for very large files, and it lets a client perform many sequential operations on the same chunk with only one initial request to the master, which fits the sequential, append-heavy access pattern the system was built for.
A specific trade-off: relaxed consistency in exchange for simplicity and throughput
GFS does not give applications the strong, always-consistent view of a file that a traditional file system promises. Instead, after a sequence of successful mutations, a region of a file is only guaranteed to be 'defined' — meaning all clients see the same data, reflecting one particular mutation in full — when that region was written by a single mutation that succeeded without interference from concurrent writers; concurrent writes to the same region can leave it merely 'consistent' (all clients see the same bytes) but 'undefined' (those bytes may be an unpredictable mix of the concurrent writers' data), and a failed mutation can leave a region inconsistent, with different chunk replicas holding different data. To make this workable in practice, GFS singles out one write pattern for a stronger guarantee: record append, an operation where the client does not choose the offset at all — GFS picks an offset and appends the record there atomically, at least once. This favors concurrent producers (for example, many worker processes appending results to one shared output file) at the cost of allowing occasional duplicate records or padding gaps, which applications are expected to tolerate, for instance by embedding checksums or unique identifiers in each record so a reader can detect and skip duplicates or padding itself. The authors accepted this weaker contract deliberately, judging that most of their applications — built together with the file system — could be written to cope with it, in exchange for avoiding the coordination cost that a stronger guarantee would impose at their scale.
Staying up under constant failure: leases, heartbeats, and a resilient master
Because component failure is routine at this scale, GFS pushes fault tolerance into nearly every mechanism rather than treating it as an add-on. For writes, the master grants one replica of a chunk a temporary 'primary' lease (60 seconds, renewable while mutations are ongoing); the primary is responsible for picking a serial order for concurrent mutations and forwarding that order to the other replicas, which avoids the master having to coordinate every single write. The master continuously exchanges heartbeat messages with every chunkserver; when a chunkserver stops responding, the master notices and starts re-replicating any chunks that fell below their target replica count elsewhere in the cluster, so the system heals itself without an operator intervening for routine disk or machine loss. Each chunkserver independently keeps checksums for its chunks and verifies them on every read, so silent data corruption on a single disk is caught locally rather than being served to a client as if it were valid. The master's own state — the namespace and file-to-chunk mapping — is itself made durable through an operation log replicated to multiple machines plus periodic checkpoints, and a 'shadow master' can serve stale-but-useful reads of metadata if the real master is temporarily slow or unreachable, so a single master, despite being one process, does not become a single point of total unavailability for the whole cluster.
Legacy: the master/worker, large-chunk pattern in later distributed storage systems
GFS was an internal Google system, not a product other companies could adopt directly, but the architecture it described — one relatively lightweight metadata-holding master paired with many storage-holding workers, large fixed-size data blocks, and background re-replication driven by heartbeats — became a template that reappeared widely once other organizations needed similar large-scale batch storage. The most direct example is the Hadoop Distributed File System (HDFS), an open-source system built at Yahoo! and later widely adopted, whose NameNode/DataNode split mirrors GFS's master/chunkserver split closely enough that HDFS's own documentation and its authors describe it as directly modeled on the GFS design, down to storing checksum metadata alongside each data block and replicating blocks across machines for durability. Google itself later moved beyond the original single-master design in its successor system, Colossus, to remove the scalability ceiling that came from having just one master, but the basic split between metadata service and bulk data storage that GFS popularized persisted. More broadly, the idea that a storage system's consistency guarantees and API should be chosen to match the applications actually built on top of it, rather than aiming for a generic standard, influenced how later systems justified similarly non-traditional trade-offs in their own designs.
See the ideas at work
Practice the designs that use this paper in the System Design school.