Dipperstein All articles
Experiments

My Code Reviews Were Mostly Vibes: A 2,300-Comment Autopsy

Dipperstein
My Code Reviews Were Mostly Vibes: A 2,300-Comment Autopsy

Photo: DFID - UK Department for International Development, CC BY 2.0, via Wikimedia Commons

I have a confession that will surprise nobody who has worked with me: I like leaving code review comments. Not in a malicious way. More in a “this could be a little cleaner” way, which is the software equivalent of rearranging someone’s silverware while they are still eating.

A few months ago I got curious about what I was actually saying in those comments. Not what I remembered saying. Not what I told myself I was saying. The real record. So I exported every review comment I could get my hands on from my own work repos and a handful of open-source projects, dumped them into a notebook, and started sorting.

This is the Dipperstein version of an audit: less compliance, more “oh no, that’s me.”

The dataset was messier than I expected

The final pile was 2,314 comments left between 2019 and 2025. That span matters because my jobs changed, my languages changed, and apparently so did my tolerance for three-line functions. I stripped out automated bot comments, merge-conflict notices, and anything that was just a code suggestion with no prose. I kept questions, warnings, praise, and the dreaded “nit:” comments.

Then I did what any reasonable engineer does when faced with unstructured text: I made a spreadsheet, wrote a half-broken Python script, and pretended the categories were obvious.

They were not obvious.

My first pass used eight categories: style and formatting, naming, readability, bug risk, architecture, documentation, process, and praise. I hand-labeled 400 comments to get a feel for the distribution, then trained a simple classifier on the rest. The classifier was not fancy. A few keyword rules plus a logistic regression model over TF-IDF features. It was the machine-learning equivalent of a label maker, but it was enough to expose patterns.

What I actually complain about

The headline number was uncomfortable: roughly 58% of my comments were style, formatting, naming, or readability. Another 15% were questions that mostly meant “I do not understand this yet.” True bug-risk comments were about 7%. Praise was under 4%, which is its own small tragedy.

I want to be clear that style comments are not worthless. Consistency matters. Naming matters. Readability is a maintenance feature. But 58% is not a code review strategy. That is a personality trait with a keyboard shortcut.

A few specific patterns showed up over and over:

The classifier also found a category I did not plan for: comments that were really about my own uncertainty. Things like “can we simplify this?” or “is this necessary?” often meant “I have not loaded enough context into my brain yet.” That was not a bug in the code. It was a bug in my review process.

The methodology was not exactly pristine

Before I get too proud of this experiment, here are the ways it was held together with tape:

First, the export was incomplete. Some comments lived in repos I no longer have access to. Some review threads were deleted. Some conversations happened in chat and never made it to the pull request at all. So this is not a complete record of my reviewer behavior. It is a biased sample of the comments that survived.

Second, my categories overlapped. A comment like “rename this and split the loop” is naming, readability, and architecture depending on my mood. The classifier had to pick one primary label, which flattened nuance.

Third, sarcasm and tone do not survive a bag-of-words model. A comment that read as dry to me may have read as hostile to someone else. I did not measure that, and I should have.

Fourth, I never tracked severity. A blocking bug and a preference about blank lines both counted as one comment. That probably inflated the style numbers and definitely made the whole dataset look more dramatic than it was.

The rubric I use now

After staring at this data, I changed how I review. Not dramatically. I am not out here writing poetry in pull requests. But I now run every comment through a short internal checklist before I post it.

1. What is the risk if we merge this as-is?

If the answer is “none, I just prefer it differently,” I label it as a preference and move on. If the answer is “this will page someone at 2 a.m.,” I lead with that. The comment should make the risk obvious before the suggested fix.

2. Is this the first time I have said this in this thread?

Repeating the same nit across six files is not thoroughness. It is a notification storm. I comment once, note that it applies elsewhere, and trust the author to sweep for it.

3. Would I want to read this after a long day?

This is the tired-teammate test. If the comment needs three paragraphs of preamble, it probably needs a direct question instead. “Can we move this into the existing helper?” beats “I was thinking about the broader architecture and how this might interact with the helper we discussed last quarter...”

4. Am I asking a question or making a demand?

Questions are good. Passive-aggressive questions are not. If I genuinely do not understand, I say so. If I want a change, I say that. The worst review comments are the ones that make the author guess which one it is.

5. Did I say anything positive?

Not as a participation trophy. Just as calibration. If the code is solid, say it. If the approach is clever, say it. Otherwise my review history turns into a list of everything that was slightly wrong, which is a terrible way to document a collaboration.

What the numbers did not fix

I would love to tell you that after this audit my bug-catch rate went up and my style comments went to zero. That would be a better ending. The truth is messier. I still leave naming comments. I still get annoyed by inconsistent formatting. I still have opinions about early returns.

But I now know the shape of my own reviewing brain. I know that most of my comments are about taste, not correctness. I know that my confidence and my verbosity are not the same thing. And I know that a simple classifier, a messy export, and an afternoon of honest labeling can tell you something a year of self-assessment cannot.

The semicolon was never the problem. Neither was the blank line. The problem was me treating every review as a chance to explain my preferences to a captive audience. These days I try to spend that attention on the things that actually break.

All Articles

Related Articles

The Leap Second Is a Legacy System, So I Broke My Homelab on Purpose

The Leap Second Is a Legacy System, So I Broke My Homelab on Purpose

The Night Sky Has No Search Bar: A 2 A.M. Wayfinding Experiment

The Night Sky Has No Search Bar: A 2 A.M. Wayfinding Experiment

Before You Refactor That Monolith, Thank a 1947 Warehouse Manager

Before You Refactor That Monolith, Thank a 1947 Warehouse Manager