Ad click aggregation

01 · CONTEXT AND REQUIREMENTS

What a click aggregator needs to do

An ad click aggregator collects every click users make on ads on a site or app and delivers performance metrics to advertisers. There are two functional requirements: send the user to the advertiser's site, and let the advertiser query clicks over time, with a minimum granularity of one minute. It is a good exercise in real-time analytics architecture.

The reference scale is ten million active ads and a peak of ten thousand clicks per second. The non-functional requirements that drive the design are: low-latency advertiser queries, fault-tolerant collection that does not lose clicks, data as close to real time as possible, and idempotent counting that never counts the same click twice.

02 · THE CLICK PATH

Redirect and record, the naive version

There are two ways to redirect. On the client, the ad already carries the destination URL and the browser navigates on its own while a parallel call records the click; sophisticated users can copy the URL and go straight there, bypassing tracking. On the server, the click arrives at our endpoint, which records the event and responds with a 302 redirect to the advertiser; it is slightly more complex, but guarantees that every click is seen.

1. The user clicks the ad and the browser calls the click service, which will record the event and respond with the redirect.

2. The service writes each click as a row in the single events database.

3. When the advertiser asks for metrics, the same database has to group all the rows in the period, and that is where the math stops working.

At peak scale, this design does not hold: the database becomes a bottleneck and the grouped query gets slow, breaking the low-latency requirement. The way out is to separate whoever receives the clicks from whoever answers the queries, and to aggregate before the advertiser asks.

03 · FROM BATCH TO STREAM

Batch aggregation and why it is not enough

The natural improvement writes raw events to a write-optimized event store and lets a periodic job, for example in Spark, read those events and write counts per ad and per minute to an analytical database. OLAP databases use columnar storage, which is very fast for aggregations. A time-series database would serve simple queries, but with millions of ads and several analysis dimensions, OLAP is a better fit.

1. The click passes through the load balancer and reaches a click processor instance.

2. The processor writes the raw event to the event store, which absorbs many writes.

3. At each interval, the aggregation job reads the events accumulated in the event store.

4. The job aggregates by ad and by minute and writes the counts to the analytical database.

5. The advertiser queries only the analytical database, which already has everything aggregated, and the response is fast.

The cost is delay: the advertiser always sees data that is a few minutes old. And during a burst of clicks, the next batch processes a much larger window and may take longer than the interval itself, causing delays exactly when the advertiser most wants to look. A queue in front of the event store would help with write spikes, but would not solve latency.

04 · REAL TIME

Stream, windows, and event time

The alternative processes events as they arrive, instead of waiting for a batch job: the click processor writes to a stream, such as Kafka or Kinesis, and a stream processor, such as Flink, aggregates in memory and writes the result to the analytical database. Flink aggregates by event time, when the click actually happened, rather than by the time the event arrived, to get the count right even with out-of-order events. Watermarks track the progress of event time and handle late events.

1. When the ad is displayed, the placement service gives the browser a signed impression ID, unique to that display.

2. On click, the browser sends the ID to the click processor, which verifies the signature and checks the impression cache.

3. If the click is new, the processor writes it to the stream first and only then records the ID in the cache; it then responds with the redirect.

4. The stream aggregator reads the stream, aggregates by ad and by minute, and writes to the analytical database.

5. In parallel, the stream archives the raw events to the data lake, and batch reconciliation re-aggregates them and corrects the analytical database if there is a divergence.

6. The advertiser queries the analytical database and sees near-real-time metrics.

The window can close at the minute boundary while partial results are flushed every few seconds, so the current minute appears incomplete, as expected, until it ends. It is also far more realistic to shrink the Flink window than to increase the frequency of Spark jobs, because of the overhead of each job.

A plain Kafka consumer with an in-memory counter does work in a simpler scenario. But event-time windows, watermarks, and state recovery built by hand on top of it are feasible and error-prone, and that is what the stream processor delivers out of the box.

05 · SCALE AND HOT PARTITIONS

Scaling each stage and the viral ad

Each stage scales horizontally: the processor behind the load balancer, the stream through partitions, and the aggregator with one task per partition. Partitioning the stream by ad identifier, the AdId, is natural, since the partitions stay independent and all events for an ad land in the same one.

The remaining problem is the hot partition: an ad that just went viral sends all its clicks to a single partition, which gets overloaded, increases latency, and in the worst case loses data. One way out is to append a random number to the AdId in the partition key, only for popular ads, so the traffic spreads out. When writing to OLAP, Flink strips the suffix and uses the original AdId, and the sums from the partitions combine correctly.

06 · DON'T LOSE, DON'T DOUBLE COUNT

Retention, reconciliation, and idempotency

Losing clicks means losing money, and the stream helps: by default it is distributed and fault tolerant, and can retain data for a period even after it has been consumed. If the aggregator goes down, it comes back and rereads the stream. Since the windows are short, an outage costs at most one minute of aggregation, which is reprocessed from a known point in the stream.

1. The stream aggregator goes down in the middle of the minute, and the analytical database stops receiving new counts.

2. Clicks keep arriving at the stream, which retains them even with no one consuming.

3. The aggregator comes back, rereads the stream from a known point and re-aggregates what was missing, and the analytical database catches up.

Even so, processing failures, buggy code, or out-of-order events can leave small inaccuracies. That is why raw events also go to a data lake and a periodic job, for example a daily one in Spark, re-aggregates them. If the result diverges from what the stream produced, you investigate the cause and correct the analytical database. Combining a fast layer (Flink) with a batch layer (Spark) for correction is the Lambda architecture.

For not counting the same click twice, counting one click per user and ad is wrong for retargeting, where the same ad is deliberately shown to the same user several times. The right approach is to count one click per ad instance: the placement service generates a unique impression ID for each display, and the click carries that ID as an idempotency key.

Deduplication has to happen before the stream, because a duplicate on opposite sides of the minute boundary would count as two clicks if deduplication lived inside the aggregator. To prevent forged clicks with made-up IDs, the ID is signed with HMAC and a secret key before it goes to the browser, and the processor verifies the signature.