Netflix EVCache: memcached replicated across regions

01 · CONTEXT

What EVCache is

EVCache is a cache that is distributed, in memory and based on memcached. The name is an acronym: Ephemeral, because the stored data is short-lived, according to the TTL; Volatile, because the data can disappear at any moment, for example through eviction; and Cache, an in-memory key-value store.

According to the EVCache client announcement, the advantages of a distributed cache are faster responses than fetching from the source or the database, less load and fewer servers, because the cache serves most requests, and higher throughput of the services it protects.

The InfoQ article describes the scale of Netflix's deployment: four regions and 200 Memcached clusters, each dedicated to one application or a group of applications. The infrastructure totals 22,000 server instances and handles 30 million replication events globally and 400 million operations per second in total. The data reaches about 2 trillion items and 14.3 petabytes.

02 · INSIDE A REGION

One cluster per availability zone

In the model described in the client announcement, an EVCache App is a logical grouping of memcached instances, and the EVCache client manages the operations between the Java application and that App. Data is partitioned across the nodes by consistent hashing, the Ketama algorithm. When there are several zones, each cluster sits in one availability zone, and the client gets the list of servers from Eureka to assemble the clusters by each one's zone.

1. The EVCache client gets the list of servers from Eureka and assembles one cluster per availability zone.

2. On reads, the client always queries the local zone, which improves latency.

3. On writes, the client writes to all zones. With the data replicated, availability increases.

4. If a piece of data is lost in the local cluster due to instance failure or eviction, it can be fetched from the other cluster, and this is generally much faster than fetching from the source.

5. If the local cluster loses instances, it can be put in write-only mode and all reads go to the other zone.

A client's reads always go to the local zone, and writes go to the zones. Reading locally improves latency and writing to all increases availability. When a read has to go to another zone, the announcement says latency goes up but reliability is higher. In the case of instance loss, directing the read to the other zone avoids an impact on latency and hit rate.

03 · WHY REPLICATE ACROSS REGIONS

Avoid recomputing when the region changes

According to the InfoQ article, in a region switch (failover) having the cache data already available in the destination region guarantees uninterrupted service with minimal latency. The cost reason is specific to Netflix's case: recomputing data after a cache miss can consume a lot of CPU, and building personalized recommendations requires significant computational resources, with machine learning algorithms as a central piece. Replicating across regions reduces that recomputation.

The article states that EVCache was designed to withstand failures at several levels (instance, availability zone and region). The client is topology-aware: it knows the physical and logical location of the servers.

04 · THE REPLICATION SERVICE

From the client to the cache in the other region

The article describes replication as a flow in stages. The origin is the write: the application uses the EVCache client to send mutations, such as set, add, delete and touch, to the local servers.

1. The application sends the mutation to the EVCache client, which applies it on the local servers.

2. EVCache also sends Kafka an asynchronous event with metadata only: key, TTL and creation timestamp, without the value.

3. The reader reads the messages from Kafka and does a read on the local EVCache. It can filter out unwanted messages, such as those with a very short TTL.

4. The reader makes a cross-region call to the writer in the destination region, carrying metadata and the value.

5. The writer writes the key-value pair to the destination EVCache. If that region starts serving traffic, the data is already available for reading.

6. If any stage fails, the failed mutation is captured and sent to an SQS queue, to be retried later.

A single Kafka cluster serves the replication of more than 200 EVCache clusters. Each EVCache cluster is represented by a topic, partitioned according to event volume. In the reader service there are distinct consumer groups, and each group targets a different region.

1. A single Kafka cluster concentrates the replication. Each EVCache cluster is a topic, divided into partitions according to event volume.

2. The readers are organized into consumer groups. Each group is a set of nodes, and each node reads several partitions, which allows parallel processing.

3. Each consumer group targets a different region, and this is how the data is replicated to several regions.

Before crossing the region, the reader converts the key, the metadata and the value to JSON, and the send to the writer is done over REST.

Failures at any stage do not lose the mutation: it goes to an SQS queue, and the replication service monitors the queue and reprocesses what failed through the same flow. According to the article, this retry mechanism guarantees that all mutations are eventually processed.

05 · CLIENT OR SERVER

Who initiates the replication

Netflix chose client-initiated replication, mainly because the EVCache client is topology-aware. It knows the location of the memcached nodes, knows which are available, understands how data is distributed among them and can decide based on network latency.

06 · EFFICIENCY AND LIMITS

Fewer bytes and fewer intermediaries

The first tweak was to group messages into batches and compress the batch before the reader sends it to the writer. The article reports a 35% reduction in network bandwidth usage.

The second tweak was to remove the Network Load Balancers (NLBs) that managed communication between reader and writer, because that arrangement generated additional network transfer cost. In their place, Eureka DNS is used: the reader obtains the writers' IP addresses and does the routing itself. The article reports a 50% drop in transfer cost, with predictable latencies. Daily NLB usage, around 45 GB/s summed across regions, fell to under 100 MB/s, and the NLBs remain in the architecture as a fallback option.

1. In the initial design, an NLB managed communication between reader and writer, and that arrangement generated additional network transfer cost.

2. Afterward, the reader began obtaining the writers' addresses through Eureka DNS and doing the routing itself.

3. The NLBs remain in the architecture only as a fallback option.