Design for the Engineer Who Got Woken Up: A Field Guide to 2 A.M.-Proof Systems
Photo: dark server room at night glowing lights, via thumbs.dreamstime.com
There's a special flavor of confusion that only exists at 2 A.M. You know the one: phone buzzing on the nightstand, half your brain still dreaming, and an alert on your screen that says something like ERROR: operation failed with no other context. Which operation? Failed how? Failed when? Your brain at that hour has roughly the processing power of a toaster, and your system just handed it a riddle.
After a decade of on-call rotations across a few different shops — a startup, a mid-size SaaS company, and one place with a genuinely heroic amount of legacy Perl — I've come to believe that we evaluate our engineering practices against the wrong persona. We review code as our fresh, caffeinated selves. We design runbooks sitting in a comfortable chair with a full night's sleep behind us. Then we ship the whole thing to a version of ourselves that doesn't exist in the office: the exhausted responder, woken out of a dead sleep, holding a phone with one eye closed.
If your design can't survive that version of a human, it doesn't matter how elegant it looked in review.
Three Postcards From the Incident Trenches
Every on-call veteran has a collection of these. Here are three of mine, lightly anonymized.
Postcard one: the vague alert. Early in my career, our monitoring sent a page that just said "Service unhealthy." That's it. Not which service — the alert body assumed you'd infer it from which alert rule fired, and the alert rule was named after an internal ticket number. I spent twenty minutes just figuring out what was broken before I could even start asking why. Twenty minutes of pure overhead, at the hour when every minute feels like a hour.
Postcard two: the cryptic stack trace. We had a distributed job runner that failed with a stack trace eighteen frames deep, ending in a null pointer in a class named AbstractHandlerFactoryImpl. The actual root cause — an expired certificate — appeared nowhere in the output. A rested engineer with the architecture diagram open might have traced it. The 2 A.M. version of me rolled the service and went back to bed, which meant it failed again an hour later.
Postcard three: the config drift. A staging-only flag had leaked into production config three weeks earlier. Nothing alerted on it, because drift detection was considered a "nice to have." When the related feature finally fell over, the responder spent the incident debugging code that was fine, because the diff between prod and the last known-good deploy was four thousand lines of noise. There was no way to answer "what changed recently?" in under an hour.
Notice what these have in common: none of them are hard engineering problems. They're all failures of empathy toward a future, impaired reader.
The 2 A.M. Test
Here's the filter I now apply to everything I ship, and I'd encourage you to steal it: imagine the least capable legitimate user of your operational artifacts is you, at your worst. Then check the following:
- Does every alert answer "what is broken and where?" in the first sentence? If the responder has to open a second system to identify the failing component, the alert failed. Alert names should be English, not ticket numbers.
- Do your error messages state the intent that failed, not just the mechanism? "Failed to renew auth certificate for billing-sync, next rotation in 30 days" beats a null pointer in a factory class. Log the why near the what.
- Can a responder answer "what changed recently?" in under two minutes? Config drift detection isn't a nice-to-have. Neither is a deploy marker in your logs. The question everyone asks at 2 A.M. is "did we change something?" — make it cheap to answer.
- Is there exactly one obvious safe action? Runbooks written as decision trees with three branches and no default are traps. When in doubt, the documented action should be "page the service owner," not a guessing game.
- Does your system fail loudly in ways a human can perceive? Silent degradation is the worst failure mode for tired people, because tired people trust the quiet. If the queue is backing up, something should say so before the pager does.
None of this is novel. That's kind of the point — we all know this stuff, and we deprioritize it anyway, because it doesn't show up in code review or in demos. The 2 A.M. test gives it a forcing function.
A Note on the Humans, Too
One last thing, because I've watched too many good engineers burn out on rotation: the test applies to your rotation design, not just your code. If your alerts routinely fire at hours when nobody can act on them, that's a design smell. If your rotation is small enough that people are afraid to hand it off, that's a design smell too. The kindest thing you can do for the 2 A.M. version of your teammate is reduce the number of 2 A.M.s.
Write your software for the person you are at your worst. The well-rested version will thank you — briefly, over coffee — and then forget about it, which is exactly what good operational design earns: the privilege of being ignored.