Dipperstein All articles
Deep Dives

My Well Pump Is the Worst Database I've Ever Administered

Dipperstein
My Well Pump Is the Worst Database I've Ever Administered

Every engineer I know has a mental model for capacity planning that goes something like: estimate load, add 30% headroom, page someone if it spikes. Nature doesn't work that way. Nature works with hard ceilings, slow refills, and a failure mode that involves your lawn turning brown and your shower sputtering at 6 a.m.

Last spring I wired up a residential well on my property in Indiana — pressure transducer, flow meter, a Raspberry Pi in a plastic tote in the well house — and logged everything for twelve months. I went in expecting a fun hobby dataset. I came out with the best capacity planning lecture I've ever attended, delivered by a hole in the ground.

The Rig, Because Someone Will Ask

The setup is embarrassingly cheap by server standards:

Total cost: about $180, most of it in weatherproofing I got wrong the first time. If you've ever instrumented anything, none of this is exotic. The exotic part is the system being measured.

The Schema of an Aquifer

Here's where the analogy stopped being cute and started being useful. An aquifer behaves like a database in ways that are almost rude in their precision:

The water table is your storage layer. It has a finite capacity. You don't get to negotiate with it.

Rainfall is your ingestion pipeline. But here's the catch — recharge isn't instantaneous. Water takes weeks to percolate down through soil. My data showed rainfall events correlating with water table recovery on a lag of 18 to 40 days depending on soil moisture. That's a write-ahead log with a terrible flush interval.

The pump is your read workload. Every shower, laundry load, and, crucially, every sprinkler run is a query. Queries draw down the table. Too many concurrent queries and you hit the equivalent of connection exhaustion, except the error message is a gurgle and sediment in your coffee maker.

Drawdown is your latency graph. When the pump runs, water level in the well bore drops fast, then recovers as the aquifer flows in. The recovery rate is literally a throughput measurement of the surrounding geology. I could graph "queries per second sustained before degradation" for my yard. It's about 9 gallons per minute, forever, no matter how big my pressure tank is.

A Year of Data, Or: The Drought Was Very Instructive

The boring months taught me nothing. Spring and fall, recharge kept pace with demand and the water table idled around 22 feet. Fine. Databases in staging are also boring.

Then came late July. We hit three weeks with basically no rain, right when lawn irrigation across the county peaked. My water table dropped about four feet in a month. My neighbor — a retired farmer who has forgotten more about water than I'll ever know — mentioned his well had "been acting tired." Meanwhile, the county extension office posted a notice about declining water tables in the watershed.

Here's the thing that should make every SRE uncomfortable: nothing scaled. There was no autoscaling. There was no bursting. There was no region failover. Demand went up and supply went down simultaneously, which anyone who has run a flash sale on under-provisioned infrastructure will recognize instantly. And the recovery time after rain returned? Weeks. Not minutes. The system had a recharge SLA measured in seasons.

I graphed it out of curiosity: the drawdown curve during the drought looked almost exactly like a connection pool exhaustion graph during a traffic spike. Same shape. Same slow tail on recovery.

Four Lessons for People Who Provision Servers

1. Hard limits change how you design. Because the well physically cannot produce more than ~9 GPM, we changed behavior: laundry at night, drip irrigation instead of sprinklers, no two big draws at once. This is just load shedding and request batching, forced by physics. Most cloud engineers have never experienced a resource that cannot be bought out of its limit, and it shows in their architectures. Try designing one system as if scaling were illegal.

2. Monitor the leading indicator, not the alarm. By the time the pump sucks air, you're way past the point of graceful intervention. The useful signal was the rate of change of the water table — drawdown velocity during high demand. In server terms, that's not CPU at 90%; that's the slope of your queue depth. Watch derivatives, not thresholds.

3. Recharge is a real constraint, not a footnote. Databases have this too. Replication lag, backup windows, cache warming after a failover — these are all recharge processes with their own timelines. Capacity plans that only count peak demand and ignore recovery time are how you end up degraded for a week after a five-minute incident. Ask me how I know (it was a Postgres failover, and also it was August).

4. Seasonality is load you can't flatten. My demand curve is spiky in summer, and no amount of clever scheduling fully smooths a drought. Some workloads are like this too — tax season, Black Friday, the first warm day in Chicago when every cyclist's app lights up. If your capacity plan assumes uniform load, it's a fiction with extra steps.

The Punchline

I still work on systems with autoscaling, and I'm grateful for it. But spending a year as the reluctant DBA of a hole in the ground rewired something. The well taught me to ask the question the cloud lets you forget: what is the actual, physical ceiling of this system, and what happens the day we meet it?

Nature runs its infrastructure with no headroom, no horizontal scaling, and a support ticket queue measured in geology. It's the worst database I've ever administered.

I wouldn't trade the education for anything.

All Articles

Related Articles

The Kettle, the Ladle, and the Compiler: A Soaking-Hot Tour of Type Strictness

The Kettle, the Ladle, and the Compiler: A Soaking-Hot Tour of Type Strictness

The Big Dipper Doesn't Move, But My Tripod Sure Did: A Star Tracker Built Out of Junk Drawer Physics

The Big Dipper Doesn't Move, But My Tripod Sure Did: A Star Tracker Built Out of Junk Drawer Physics

Burying a Library With Dignity: Confessions of an Open-Source Undertaker

Burying a Library With Dignity: Confessions of an Open-Source Undertaker