Papers
Reducing the Scope of Impact with Cell-Based Architecture
Robisson Oliveira · AWS Well-Architected whitepaper (published September 20, 2023), 2023
The problem: a single blast radius and the idea of the bulkhead
The whitepaper starts from an observation: organizations need to be increasingly resilient, and in multi-tenant cloud environments customers still expect the experience of a single-tenant environment. The guide proposes bringing to the workload architecture level the same fault isolation that AWS applies in Availability Zones and Regions, expanding the Well-Architected best practice of using bulkhead architectures to limit the scope of impact. The metaphor comes from ships: vertical partitions create watertight compartments, so a breach in the hull stays contained in one section. In complex systems, fault isolation boundaries restrict the effect of a failure to a few components, and whatever is outside the boundary is unaffected. The document contrasts this with a typical three-tier application, in which a failure or a change hits 100% of customers. The proposal is, instead of building services as single-image systems, to split them internally into cells and put a thin layer in front to route traffic to the right cell. According to the text, AWS service teams have used cell-based architecture for over a decade, and it can be zonal, regional or global. The guide states that this style gives more fault isolation, predictability and testability, properties that are fundamental for workloads that require extreme levels of resilience.
Components: cell router, cell, and control plane
Cell-based architecture has three components. The cell is a complete instance of the workload, with everything it needs to operate independently; for example, if the application is an Application Load Balancer, a few EC2 instances and an Amazon RDS database, all of these elements belong to cell 1, and a cell 2 requires another deployment of all three. Ideally the cell does not know about the others, does not share state, and makes no calls between cells or uses shared resources such as databases or S3 buckets; the guide even encourages separate AWS accounts. Dependencies between cells quickly eliminate the benefits, so they should be minimal or only transient. The cell router is described as the thinnest possible layer, with the sole responsibility of routing requests to the right cell, presenting a single endpoint to clients. The control plane handles administrative tasks, such as provisioning and removing cells and migrating customers between them. The text reminds us that this does not necessarily mean doubling or tripling the infrastructure: an application with 30 hosts can stay at 30 hosts, just grouped into cells behind a router. Partitioning uses a partition key, such as customer or resource ID, aligned with the service's grain. With 10 cells serving 100 requests, a failure in one cell leaves 90% of requests unaffected.
Control plane, data plane and static stability
The guide separates the control plane from the data plane, terms borrowed from the networking world. The control plane provides the administrative APIs to provision, move, migrate, update, remove, deploy and monitor cells. The data plane runs the service's core function, together with the cell router. If there are five cells and the number of users grows, the control plane provisions a new cell and tells the router where to send traffic; after that, the router and the cell just do the data plane's work. The text invokes static stability, recommended in the Well-Architected Framework (REL11-BP04, rely on the data plane and not the control plane during recovery): the system keeps working even when a dependency is impaired. According to the text, control planes are statistically more likely to fail than data planes; even so, resources that are already provisioned remain available even if creating, changing or deleting is compromised. The guide also states that control planes are designed to fail rather than corrupt or give incorrect information (CP in the CAP theorem), while data planes generally prefer AP, trying to stay available even with stale information. As an example, a router that loads routes into memory from an S3 bucket keeps directing traffic even if the control plane, S3 or a zone are unavailable.
Partitioning, routing and cell size
The partition key should follow the grain of the service and be easily accessible in most API calls. The guide warns that CustomerID looks like a good candidate, but you need to think about huge customers that don't fit in a single cell; one strategy is to include a second dimension in the key. Interactions that cross cells, such as scatter-gather, should be the minority; one approach is to have them go back through the router instead of allowing direct calls between cells. Among the algorithms listed, with no specific recommendation, are full mapping (flexible, but with a critical dependency on the table, a requirement for read-your-writes consistency, and a lot of state), prefix or range mapping (reduces cardinality, but favors hot cells), naive modulo (uniform distribution and minimal state, but requires rebalancing everything when the number of cells changes), and consistent hashing (less movement, but possible imbalance). For any approach except full mapping (which already offers this natively), an override table is recommended to force keys onto specific cells. The router should be as simple and horizontally scalable as possible, avoiding complex business logic, and it is the only component holding the state of all cells, so it is a single point of failure that must also be built in a cellular way. On size, three forces pull against each other: large enough to fit the biggest workloads, small enough to test at full scale, and large enough to have economies of scale. Smaller cells reduce the impact, for example 1% per cell with 100 cells versus 10% with 10, but increase the number of replicas to operate.
Trade-offs, wave-based rollout, and best practices
The guide lists benefits: scaling horizontally through cells of a known maximum size, a smaller blast radius, greater testability (it is reasonable to simulate the largest load that fits in a cell, which should match the largest load of a single customer), higher MTBF, lower MTTR and higher overall availability. A natural conclusion would be that availability ends up the same as a monolith's, since n cells have n times as many failure events, each with 1/n of the impact; but the higher MTBF and lower MTTR mean less frequent and shorter failures per cell and, therefore, higher overall availability. The question posed is whether it is better for 100% of customers to have a 5% failure rate or for 5% of customers to have a 100% failure rate. The stated drawbacks are greater complexity, higher infrastructure cost, specialized tooling and operational practices, and the need to invest in a routing layer; for this reason, the architecture does not suit every workload. Cells were also not designed as failover domains or to mitigate dependency failures. For deployment, the key point is to deploy in waves, cell by cell or in sets; the first cell can be a canary, and at the first sign of failure you can roll back; an automated CI/CD pipeline from the start and cell-aware observability are essential. Among the best practices: treat the current stack as cell zero (put the router above it and shift traffic over gradually according to the partitioning strategy), start with multiple cells (which brings early experience operating in this environment and reduces surprises) and with a migration mechanism from day one, and do a failure mode analysis of each cell.
See the ideas at work
Practice the designs that use this paper in the System Design school.