Dipperstein All articles
Experiments

The Leap Second Is a Legacy System, So I Broke My Homelab on Purpose

Dipperstein
The Leap Second Is a Legacy System, So I Broke My Homelab on Purpose

My homelab is a small rack in the basement: a mini PC running Proxmox, a Raspberry Pi doing DNS and metrics, a cheap UPS, and enough blinking LEDs to make me feel like a sysadmin. It is also where I test ideas that would get me side-eyed in production. Last weekend, the idea was a leap-second fire drill.

This is not an abstract concern. Leap seconds have been inserted into UTC since 1972, and the last one landed in 2016. Metrologists have since agreed to stop adding them by 2035, but that is a future promise, not a current behavior. In the meantime, every cron job, log pipeline, and monitoring rule that treats UTC as a simple, monotonic counter is carrying a small assumption with sharp edges.

So I built a fake one.

The setup: a time warp in a closet

I did not touch the real clock on my hypervisor. That is how you turn a fun experiment into a Tuesday-night recovery project. Instead, I ran two parallel tracks.

The first track used containers with libfaketime to inject a synthetic 23:59:60 into a handful of test jobs. The second track used a tiny Python NTP server that advertised a leap-second flag to a separate subnet of test VMs. I wanted both the application-level weirdness and the protocol-level weirdness, because real leap seconds show up in both places.

My test workload was boring on purpose:

The whole thing took an afternoon to assemble, mostly because I spent forty minutes looking for a spare Ethernet cable. The experiment itself was less about precision and more about watching assumptions fail in public.

Failure one: cron ran the same job twice

Cron is the oldest teenager in my house. It has been around forever, it does not explain itself, and it has strong opinions about time. During the fake leap second, my backup rotation job ran once at 23:59:60 and then again at 00:00:00. Same job, same archive, two runs.

In my lab, that meant a duplicate summary email and a second rotation file with a timestamp that looked like it came from a different universe. In production, that pattern is how you get duplicate invoices, double-sent notifications, or a batch job that consumes the same queue twice.

The fix is not to trust cron to be idempotent. It is to make the job idempotent. I added a lock file keyed to the logical date of the run, not the wall-clock minute. The second run noticed the lock and exited. Boring, effective, and deeply unglamorous.

Failure two: logs arrived out of order

My log pipeline did not crash. It just got confused. A few entries from the leap-second container carried timestamps that appeared earlier than entries that had already been indexed. Loki accepted them, but the dashboard queries sorted by time and produced a small negative duration on one panel.

That panel was supposed to show how long a batch took. For about ninety seconds, it showed a job finishing before it started. I laughed, then I stopped laughing, because that is exactly the kind of thing that wakes you up at 2 a.m. when the panel is attached to a customer-facing SLA.

Timestamps are not ordering. They are metadata that happens to be useful for ordering most of the time. If the order matters, you need a sequence number, a monotonic clock, or an event ID. I added a simple sequence column to the test log stream and a note in my runbook: sort by sequence first, timestamp second. It is not elegant, but it survives a clock that hiccups.

Failure three: the alert fired twice

My alert rule used a for: 5m clause. During the drill, the rule evaluated twice against overlapping windows and fired two notifications for the same slow job. The second one arrived after the job had already recovered, which made it look like a new incident.

This is a classic leap-second side effect: the same underlying event gets counted twice because the time window is not as stable as you think. The fix in my lab was to add a deduplication key based on the job ID and the logical run date. The fix in a real system is usually the same, plus a little humility in the alert text.

I also learned that my dashboard was doing math on wall-clock timestamps when it should have been using monotonic elapsed time. time.Since in Go, CLOCK_MONOTONIC in Linux, or a monotonic counter in your metrics library will not save you from every problem, but they will keep durations from going negative.

What I would actually ship

The leap second is a legacy system. It exists because we wanted civil time to track Earth's rotation, and Earth is not a very good API. The fact that it is rare does not make it safe. Rare events are exactly where your assumptions hide.

Here is the short list I now use for any time-sensitive service, even a tiny one:

The drill did not break my homelab. It broke my confidence in a few convenient assumptions, which is cheaper. The backup job still runs, the dashboard still renders, and the UPS still beeps when I plug in the vacuum. But now there is a little note in my runbook that says: UTC is a distributed system. Treat it like one.

All Articles

Related Articles

The Night Sky Has No Search Bar: A 2 A.M. Wayfinding Experiment

The Night Sky Has No Search Bar: A 2 A.M. Wayfinding Experiment

Before You Refactor That Monolith, Thank a 1947 Warehouse Manager

Before You Refactor That Monolith, Thank a 1947 Warehouse Manager

I Ghosted My Own Codebase for 90 Days: Notes From a One-Man Bus Factor Experiment

I Ghosted My Own Codebase for 90 Days: Notes From a One-Man Bus Factor Experiment