Five fields and a command
0 2 * * * /opt/certs/renew-all
Read left to right, the five fields are minute, hour, day of month, month and day of week, and a star means "any": at minute 0 of hour 2, on every day. For two years this one line, on one server, has kept the certificates of a hosting platform's 4,000 customer domains current. Each certificate is valid for 90 days, and every night at 02:00 renew-all asks the certificate authority for a new one for each domain whose current certificate expires within 30 days, and installs it.
That line is a scheduled job: work started by the clock instead of by a request. Cron is the Unix daemon that reads such lines, the cron expression plus a command, and starts each command when the clock matches. Nobody waits for the result, so nobody is waiting to notice when it goes wrong. Cron reads the clock in the server's time zone, here UTC, so 02:00 is one unambiguous moment; a job meant for 09:00 in several countries, or one that must survive a daylight-saving change, has the separate problem worked through in Time zones, DST & recurring events.
The 02:00 that never came
In March the server is down from 01:40 to 03:10 for a kernel upgrade. When it comes back, cron waits for the next 02:00. The missed run is gone, and nothing records it: plain cron compares the clock with its lines and keeps no history. Tools built for this keep one. anacron records when each daily job last ran and starts it late if a day was missed, a systemd calendar timer (OnCalendar=) with Persistent=true does the same, and a Kubernetes CronJob starts a late run unless it is later than an optional startingDeadlineSeconds, after which that run counts as missed; if its controller counts more than 100 missed schedules, it starts no late run and logs an error.
Decide per job: catch up (run what was missed) or skip (wait for the next tick). For renew-all, skipping a night is harmless, because each run renews everything inside the 30-day window, so the next run does the missed one's work too; after a three-day outage, three catch-up runs in a row would be wasted, and one is enough. A job that bills each day's usage is the opposite. Skip Tuesday and Tuesday is never billed, so every missed day needs its own run, which none of the catch-up tools above promises: the job should find the days not yet billed, and each run should name the day it covers instead of reading "yesterday" off the clock.
Still running when the next one starts
In May the certificate authority has a bad night. Its API answers some requests after minutes instead of seconds, and renew-all, which never set a timeout on anything, is still going at 02:00 the next night. Plain cron does not check: it starts a second renew-all next to the first. Now two processes walk the same list and request certificates for the same domains, from a provider that is already struggling and may limit how many certificates one account can request.
Schedulers that handle this let you set an overlap policy per job: allow the new run alongside the old one, skip the new run while the old one is still going, or stop the old one and start the new one. Kubernetes calls these Allow (its default), Forbid and Replace. On a single machine, starting the command through flock -n with a lock file gives the skip behaviour: the second run cannot take the lock and exits at once.
Only Replace ends a hung run, and only when the next one is due. Without it, a long-running job with no timeout leaves two outcomes: the next run either piles on top of the stuck one (plain cron) or is skipped behind it (no-overlap policy). With skip, a run stuck since May means no certificate is renewed until someone notices. So give the whole run a time limit shorter than the interval between runs, and each call inside it a timeout of its own, so that a stuck run ends and the next one starts clean. Pick the limit from how long healthy runs take, the way Timeout picks a number from measured latency, not from a guess.
Off the box, and fired twice
One server is one point of failure, and nobody else can see its crontab. The platform moves renew-all to a scheduler service: a managed cron trigger, a Kubernetes CronJob, or a service of its own that keeps job definitions in a database. It should start the job on a worker, record each run with its start time and outcome, and apply the missed-run and overlap policies above; a managed trigger may leave some of that to you. Run it on two nodes so that one can fail, and both nodes see 02:00 and fire, unless they agree on who fires, the problem Distributed locks & leader election works through.
The scheduler records every run, and the job reports each success. An alert fires when no success has arrived for 26 hours.
The move brings its own failure. A scheduler that crashes after starting a run but before recording it cannot tell, when it comes back, whether that run happened, and starting it again is the safer guess. So scheduler services generally fire at least once: a run may occasionally fire twice. Not even that is promised everywhere: the Kubernetes documentation says a CronJob creates a Job about once per scheduled time, in some circumstances two or none, so jobs should be idempotent: running one twice leaves the same result as running it once, the property Idempotency & exactly-once illusions examines in general.
renew-all is nearly there by accident: a domain renewed at 02:00 no longer expires within 30 days, so a second run a minute later skips it. The gap is two runs that read the list before either has installed anything. Narrowing it means making each domain's renewal a conditional step, marked as in progress in the database only if no other run holds an unexpired mark, with an expiry so a crashed run's mark lapses. A run that outlives its mark, or crashes after the authority issued a certificate, can still lead to a second request; for renewal that only wastes a certificate. Each firing should stay one unit of work like this, safe to start again from the top; a run of several steps that must resume where a crash left it is a durable workflow (Durable workflows: orchestration that survives restarts), and the scheduler's part is only to start it.
The run nobody missed
The worst failure makes no noise. In August the platform moves its scheduler to a new cluster, and the renewal job is left out of the list of jobs to migrate. Nothing fails, because nothing runs. The scheduler's own Health Endpoint Monitoring check stays green, because the scheduler is healthy; it just has nothing to fire. About a month later, the first certificates that should have been renewed expire, and customers' visitors get browser warnings.
An alert on failed runs would have stayed silent too, since there were none. What catches this is an alert on absence. The job reports each successful run, by pushing a last-success timestamp to monitoring or pinging a dead man's switch service, and an alerting rule of the kind described in Observability: metrics, logs & alerting fires when the last success is older than the interval plus a margin: for a nightly job, 26 hours without one pages someone.
Three policies on a Grid
The Distributed Job Scheduler challenge asks for recurring cron jobs that run with an at-least-once guarantee, spread across many workers without duplication: a job may fire twice, but its effects should not happen twice. Pick one recurring job and write three decisions beside its Scheduler: what happens to a run missed while the scheduler was down, what happens when the previous run is still going, and what makes a second firing harmless. Then add the alert for a job that stops succeeding, and test it against August: if the job were deleted tonight, what would page someone, and when?