Leap seconds: 2012 (Reddit, LinkedIn, Linux kernel) and Cloudflare 2017 (DNS)

01 · CONTEXT

Cloudflare's DNS and the CNAME lookup

At the turn of January 1, 2017, right at midnight UTC, a number inside RRDNS, Cloudflare's own DNS software, went negative when it should have been, at worst, zero. The effect was that some DNS resolutions for some web properties failed. Cloudflare's public post-mortem is the source for this case.

The problem only hit customers with CNAME records. In that case Cloudflare occasionally needs to query via DNS the real address of the origin server. RRDNS tracks the performance of several internal resolvers, kept in each data center to have redundancy, and picks the best-performing one in a weighted selection.

1. A query for a name configured as a CNAME reaches RRDNS.

2. RRDNS needs to find the origin address and asks one of the internal resolvers.

3. The chosen resolver performs the recursive DNS query and brings back the real address.

4. RRDNS keeps performance measurements for each resolver and uses them to decide which to pick.

02 · THE ERROR CHAIN

From a clock running backward to a panic

The root cause, according to Cloudflare, was an assumption: some code assumed the difference between two instants would be, at worst, zero. RRDNS measures a resolver's response time by subtracting two time.Now readings, and the Go function used for this does not guarantee monotonicity.

This calculation works fine if time moves forward. The resolvers were tuned to be very fast, and responding in a few milliseconds is normal. So if the clock goes back one second during a resolution, the perceived time becomes negative.

A single negative measurement would not be enough for RRDNS to think the resolver works in negative time. But it keeps several measurements and smooths them; after a few measurements the smoothed value could become negative. Later, the weighted selection code received that value, and Go's random-draw function panics with a negative argument.

1. During a fast resolution, the clock goes back and the difference between the two readings comes out negative.

2. Some resolutions write a negative value into the measurements structure during the leap second.

3. Smoothing combines several measurements and, after a few, the smoothed value turns negative.

4. The weighted selection reads the negative value and the random-draw function panics.

5. The panic is caught with recover, but some DNS resolutions fail for those using CNAME.

03 · IMPACT AND RESPONSE

Few machines, a fix in waves

The reach was limited: few machines were affected. At the peak, about 0.2% of DNS queries and less than 1% of HTTP requests to Cloudflare had a problem. The most affected machines were fixed within 90 minutes and the fix reached the whole world by 06:45 UTC.

The post includes a chart of the error rate per data center (observabilidade), which shows some locations more affected than others and the rapid drop in errors as the fix progressed. Cloudflare deployed the fix prioritizing the locations with the most errors.

1. The mitigation goes first to a canary node and is confirmed at 00:55 UTC.

2. Then it goes to a canary data center and is confirmed at 01:03 UTC.

3. The fix reaches the most impacted data center at 01:23 UTC.

4. Next it is deployed to the main data centers, starting at 01:45 UTC.

5. Errors per data center fall as the waves advance, and the impact ends at 06:45 UTC.

04 · THE FIX

Forget the measurement instead of storing the impossible

The patch made RRDNS forget the resolver's current performance and normalize again if time jumped backward. The fix prevents negative values from being written in server selection, and restarting all RRDNS servers resolved any recurrence.

With a non-monotonic clock source, the post cites as a precaution to always check whether the difference between two instants is negative. According to the post, Go did not offer a monotonic time source at the time. The company also said it was still inspecting all code for other uses of leap-second-sensitive time intervals.

05 · LESSONS

A wall clock does not measure duration

The design lesson is to separate two uses of time: knowing what time it is and measuring how long something lasted. The first depends on a clock that can be adjusted; the second calls for a clock that never goes backward. When only the first exists, code that measures duration must treat a negative result as possible.

A second design lesson is to validate at both ends: whoever produces a value and whoever consumes it can check it independently, and each check costs little.