Classic design · Senior
Distributed task scheduler
Classic interview design. Key points: job scheduling and distributed locks.
Short answer
Classic interview design. Key points: job scheduling and distributed locks. A job scheduler is a program that automatically runs jobs at set times or intervals. A job is an instance of a task and is made up of the task, the schedule, and the parameters to run it.
01 · CONTEXT AND REQUIREMENTS
What a job scheduler needs to do
A job scheduler is a program that automatically runs jobs at set times or intervals. A job is an instance of a task and is made up of the task, the schedule, and the parameters to run it.
There are two core functional requirements: schedule jobs for immediate execution, at a future date, or on a recurring basis, and let the user track their status.
The Hello Interview guide suggests assuming 10k jobs per second and lists three deep-dive questions: run the job within 2s of the scheduled time, scale to that throughput, and guarantee at-least-once execution.
02 · THE BASIC DESIGN
From the API to the worker
This article's design separates four responsibilities: the API accepts and queries jobs, the database holds the state, the scheduler decides what is due, and the workers execute. Between the scheduler and the workers sits a fila for execution.
The client asks the API to schedule a job: immediate, at a future date, or recurring.
The API writes the job to the database with the task, the schedule, and the execution parameters.
The scheduler queries the database for jobs whose time has come.
The scheduler puts each due job on the execution queue.
A worker takes the job off the queue, runs the task, and records the status in the database, which is what the user queries.
03 · AT-LEAST-ONCE EXECUTION
When the worker dies midway through the job
In a fila like SQS, if the message is not deleted before the visibility timeout ends, it becomes visible again and another consumer can pick it up.
This covers application failures, crashes, and connectivity problems, and ensures the message is not lost even if the first processing attempt fails. The timeout can be extended programmatically with ChangeMessageVisibility.
The cost is duplication: the SQS documentation does not guarantee that a message will not be delivered more than once within the visibility timeout. Cloud Scheduler also promises at-least-once execution: the job runs at least once per scheduled run and, in rare circumstances, may run multiple times for the same schedule instance.
| What it guarantees | What it does not guarantee | |
|---|---|---|
| Queue with a visibility timeout | The message becomes visible again if it is not deleted before the timeout | Does not guarantee that the message is not delivered more than once |
| Cloud Scheduler | The job runs at least once per scheduled run | May run multiple times for the same schedule instance |
| Kubernetes CronJob | Creates a Job approximately once per scheduled time | May create two Jobs or none |
04 · MORE THAN ONE SCHEDULER
The database as referee
Airflow supports more than one scheduler at the same time, both for performance and for resilience, and the high-availability scheduler makes use of the existing metadata database.
Two schedulers read the same metadata database, with no direct channel between them.
Only one scheduler at a time can be in the critical section of the scheduling loop.
The database enforces this rule with row locks (SELECT ... FOR UPDATE).
Whoever is in the critical section selects what is ready, respecting concurrency limits, and enqueues it for execution.
This is one way to coordinate several schedulers: database locks guarantee that only one of them enters the critical section.
Another way is to elect a leader by lease, the topic of the next section.
05 · LEASES AND FENCING
One leader at a time, even with pauses
A lease is a mechanism for locking a shared resource and coordinating activity among the members of a set. Kubernetes uses it to keep only one instance of a component active, with the others on standby.
The lease's expiry keeps a client that has crashed from holding the lock forever. But if a client pause lasts longer than the expiry and it doesn't notice that the lease has expired, it may make an unsafe change.
One scheduler acquires the lease and acts as leader; the other instance stays on standby.
The lease has an expiry, so that a leader that has crashed does not hold the lock forever.
The leader may pause for longer than the expiry, without noticing that the lease expired, and still try to write.
The other scheduler acquires the lease and receives a token larger than the first one's.
When the old leader comes back and writes with the smaller token, the database, which remembers the largest token already processed, rejects the write.
06 · WHEN THE TIME PASSES
Delays and overlapping runs
Kubernetes CronJob has two rules for scheduled times that pass. If the start deadline is missed, it skips that instance and the following ones remain scheduled. With the Forbid policy, if the previous run has not finished yet, the new run is skipped.
Frequently asked questions
What is Distributed task scheduler?
Classic interview design. Key points: job scheduling and distributed locks.
What does the Distributed task scheduler design cover?
It is organized in these sections: What a job scheduler needs to do, From the API to the worker, When the worker dies midway through the job, The database as referee, One leader at a time, even with pauses, and Delays and overlapping runs.
How long is Distributed task scheduler and what level is it?
It takes about 16 minutes to read and is rated Senior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.