Object storage (S3-style)

01 · CONTEXT AND REQUIREMENTS

What an object store promises

Object storage is a key-value service for large data: you choose a unique key when storing the data and use the same key to retrieve it. AWS describes S3 as a simple key-based store and says it was designed to update the whole object, unlike a file system, which supports partial update and append. This case is a good starting point for the requirements gathering of a storage system.

There is no real directory. Keys can be any string and can imitate a hierarchy, but renaming a prefix requires renaming every object it contains. For this reason, the design treats the key as a flat name and avoids operations that depend on a tree.

There are practical size limits: an object ranges from zero bytes to 50 TB, and the largest object accepted in a single PUT is 5 GB. AWS recommends multipart upload for objects of 100 MB and up, a subject for a later section.

The non-functional requirements that drive the design are durability and availability. For the standard class, AWS states a design for 99.999999999% durability and 99.99% object availability over a year. These are design goals, not contractual guarantees.

02 · OVERVIEW

Four blocks and one central separation

In broad terms, AWS describes S3 as a REST API over HTTP with four blocks: a frontend fleet with the API, a namespace service, a storage fleet full of hard drives, and a background operations fleet, which AWS calls data services, such as replication and tiering. Each block is itself made of many services: in total there are hundreds of microservices.

In this study's design, the decision that matters most is separating metadata from data. The namespace stores the key and the information about where the object is; the storage fleet stores the bytes. Each side scales and protects itself with the appropriate technique: the namespace needs fast lookup by key, and storage needs a lot of capacity with cheap redundancy.

1. The client sends a PUT with the key and the bytes to the frontend's REST API.

2. The frontend hands the bytes to the storage fleet, which chooses which disks they land on.

3. Only after the bytes are safe does the frontend record the key and the data location in the namespace. This is a choice of this design: the object only becomes visible when it exists in the namespace.

4. On a GET, the frontend looks up the key in the namespace, finds out where the data is, and reads the bytes from the storage fleet.

5. Off the client's path, the background services take care of replication, tiering, and repair.

03 · DURABILITY

Replication, erasure coding, and zones

The standard classes store each object on multiple devices, in at least three availability zones of a region, and were designed to sustain the data even with the loss of an entire zone. This is the reasoning of disaster recovery applied within a region.

There are two forms of redundancy. Replication keeps copies on different disks: it protects against the failure of a disk and allows reading from any copy, but is expensive in capacity. Erasure coding splits the object into k fragments and generates m parity fragments; as long as k of the k+m fragments are available, the object can be read, and the capacity overhead drops for the same number of tolerated failures.

1. The frontend encodes the object into fragments and distributes them across the three zones.

2. The namespace records where each fragment landed.

3. The whole of zone B becomes unavailable, and with it the fragments it held.

4. A read is still possible: the frontend fetches fragments from the remaining zones, as long as there are enough fragments to rebuild the object.

5. In the background, the repair service detects the lost redundancy and recreates the fragments on healthy disks.

Redundancy alone is not enough. The storage services quickly detect and repair lost redundancy and regularly verify data integrity with checksums.

04 · PERFORMANCE AND HEAT

Large disks, limited I/O, and hot spots

Hard drives have gained a lot of capacity, but random access is still mechanical. AWS cites about 120 operations per second of sustained random access, a value similar to 2006, when S3 launched. Capacity grows while random-access performance stagnates, and this tension shapes the design of the service.

The resulting problem is balancing I/O demand across an enormous number of disks, which the S3 team calls heat management. If many requests land on a few disks, hot spots form: nothing goes down, but requests queue up, and hot spots on individual disks create tail latency.

The answer combines redundancy and spreading. Redundancy spreads the load and allows traffic to be steered away from hot spots. In addition, even if an object is encoded across dozens of disks, different objects go to different sets of disks, so that each customer's accesses are spread over many disks.

This brings two benefits: a customer's data takes up a small share of each disk, which gives load isolation, and an individual workload can burst to a scale of disks that would be hard and expensive to build as an isolated system.

On the client side, the rule is similar to data particionamento: AWS documents at least 3,500 writes or 5,500 reads per second per partitioned prefix, with no limit on the number of prefixes in a bucket. Scaling is gradual, not instantaneous; to increase throughput, spread the keys across several prefixes.

05 · LARGE OBJECTS

Multipart upload

A multipart upload sends a single object as a set of independent parts. The parts can proceed in parallel to improve throughput, and, if the transmission of one fails, only that one is retransmitted, without affecting the others.

The flow has three steps: initiate, upload parts, and complete. Each part has a number between 1 and 10,000 and returns an ETag, and the client must record both. On completion, the service creates the object by concatenating the parts in ascending order of part number.

1. The client initiates the upload and receives a unique identifier. In this design, the service creates the upload record in the metadata.

2. The client sends the parts in parallel, each with its number, and keeps the ETag returned by each one.

3. A part fails along the way. The client resends only that part, without restarting the object.

4. The client completes the upload with the list of numbers and ETags. The service assembles the object in number order and only then does it exist as a single object.

Parts that were uploaded but not completed keep taking up space and incurring charges until the upload is completed or aborted.

06 · CONSISTENCY AND VERSIONS

Reading what was just written and not losing the past

S3 started with eventual consistency. Without strong consistency, applications needed extra code or databases to keep objects consistent. AWS began providing strong consistency for read-after-write: after a successful write, overwrite, or delete, any subsequent read gets the most recent version, and listing also reflects the change.

In a design of your own, this is reflected in the write order shown in the overview: the bytes are made safe first and the key is only recorded in the namespace afterward. That way, a read that finds the key always finds the data.

Versioning complements the protection. It is off by default and must be enabled. With it on, deleting an object inserts a delete marker instead of removing the object, and overwriting creates a new version, so the previous one can be restored.