When the Sun Is Your Sysadmin: Thirty Days of Off-Grid CI
Photo: Grendelkhan, CC BY-SA 4.0, via Wikimedia Commons
I have a bad habit of treating electricity like it is somebody else's problem. Flip a switch, rack a server, let the cloud bill arrive. So in late September 2026 I gave my self-hosted CI runner a landlord: a 100 W solar panel, a LiFePO4 battery, and a Raspberry Pi 5. For thirty days, every build had to compete with clouds, dusk, and the occasional tree branch. The experiment was supposed to be a lark. It turned into the most honest lesson in queueing I have had since I first watched a build farm melt down on a Friday afternoon.
The rig, and what it cost
This was not a science-fair project with alligator clips. I wanted something I could leave on a porch in Arizona without fear. The bill of materials:
- Raspberry Pi 5, 8 GB, with an NVMe HAT and a 1 TB SSD
- 100 W monocrystalline panel on a adjustable tilt mount
- 10 A MPPT charge controller
- 12.8 V, 50 Ah LiFePO4 battery
- 12 V-to-5 V, 5 A DC-DC buck converter
- INA219 current shunt for telemetry
- NEMA 3R enclosure, inline fuses, a gasketed cable gland, and a lot of zip ties
Total came to about $420, with the battery and panel doing most of the damage. The runner itself is Forgejo Actions on the Pi, connected to a tiny VPS that handles the web UI and job dispatch. The Pi reports battery state of charge, panel watts, and CPU temperature to a Prometheus instance every fifteen seconds. I added one rule early: if state of charge drops below 35 percent, the runner stops accepting jobs. That rule saved me more than any hardware choice.
The first brownout was a lesson in honesty
Day four gave me a build failure that made no sense. A Go test suite that had passed 200 times on grid power failed with a segfault in a dependency I had not touched. I reran it. Passed. Reran again. Failed. I did what every engineer does: blamed the code. Then I looked at the telemetry and saw the 5 V rail had dipped to 4.62 V for about 300 milliseconds during the job. The Pi had not rebooted, but the NVMe drive had briefly gone to lunch.
That is the thing about off-grid CI: it turns invisible infrastructure into a visible antagonist. On grid power, a voltage sag is a utility problem you never see. On solar, it is a test result. I added a cheap ADC to log the 5 V rail alongside the test output, and I started classifying failures into three buckets: code, infra, and sun. The sun bucket was bigger than I expected. About 11 percent of failures in the first two weeks were undervoltage events, not regressions. If I had trusted the dashboard, I would have spent a weekend bisecting a commit that was perfectly fine.
Queueing is easier when you cannot lie about capacity
A normal CI system pretends capacity is elastic. You add runners, you pay for concurrency, you let the queue grow until someone notices the bill. A solar runner cannot pretend. My battery is a token bucket with a leaky roof.
I ended up with a simple scheduler: interactive jobs (pull requests, security scans) get priority when state of charge is above 60 percent. Nightly builds and documentation jobs wait until the battery is full or the sun is strong. If the queue backs up, the runner publishes an estimated start time based on the last three days of solar yield. That estimate was wrong a lot, but it was honest, and honesty changed behavior. I stopped pushing trivial commits at 4 p.m. because I could see the sun was already low. My commit pattern got batchier. My build queue got shorter. I am not saying every team should run CI on a battery, but every team should have a queue that tells the truth about when work will start.
Backoff is not just for retries
The flakiest thing in the system was not a test. It was the weather. On day twelve, a passing cloud bank cut panel output by 70 percent for forty minutes. The runner had three jobs in flight. Two finished. One, a Rust build with a lot of linking, pushed the battery below the cutoff and got killed mid-link. The next morning I added a soft backoff: when panel watts drop below 20 W and state of charge is under 50 percent, the runner finishes current jobs but stops pulling new ones for ten minutes. That is it. A ten-minute nap. It cut brownout-triggered failures to nearly zero for the rest of the month.
I keep thinking about how much production software lacks that nap. We retry aggressively, we scale out, we treat every delay as a failure. A solar runner teaches you that sometimes the correct action is to do nothing and let the battery recover. The logs even look peaceful:
[2026-10-03T18:22:11Z] runner: queue=4 soc=0.47 panel=12W action=backoff duration=600s
That line is more useful than half the dashboards I have built.
The carbon math is humbling, not heroic
Here is where I have to puncture my own balloon. Over thirty days, the panel produced about 14.6 kWh. The runner consumed about 10.9 kWh, including the Pi, the SSD, the charge controller losses, and the inverter I used for a few test jobs. At the US average grid intensity of roughly 0.37 kg CO2e per kWh, that is about 4 kg of CO2e avoided. Four kilograms. A short drive.
The embodied carbon of the battery and panel is somewhere in the hundreds of kilograms. This experiment will not pay back its carbon debt in a month, a year, or probably the life of the Pi. That is the uncomfortable part of the off-grid compute story. The carbon cost of free compute is not zero; it is just moved to the manufacturing column. If you want to reduce CI emissions, the first move is still to run fewer jobs, cache better, and stop building the same commit three times. Solar is a nice constraint and a terrible offset strategy.
What I kept
I kept the runner. I also kept three habits that have nothing to do with solar. First, every failure now gets a tag that says whether it was code, infra, or environment. Second, the queue shows an estimated start time instead of a spinner. Third, non-urgent jobs have a real off-peak window instead of a vague hope that the runner is free.
The Pi is back on grid power now, mostly because winter sun in my part of Arizona is a part-time employee. But the scheduler still runs. When the battery is low, the queue backs off. When the sun is high, it catches up. It is the most polite CI system I have ever owned, and it learned that politeness from a 100 W panel and a battery that does not care about my deadlines.