A Reviewer Rewards Commit Messages That Explain Intent

A reviewer rewards commit messages that explain intent

A commit message earns its place in history when a future reader can understand the change without reconstructing the entire review. Fluent wording helps, but the harder skill is choosing the right facts. A model can describe a patch accurately and still miss the reason anyone needed it.

This walkthrough uses an illustrative exercise: four anonymous models receive the same small patch, and one reviewer evaluates their proposed messages. The candidates and judgments are constructed examples, not measured results from a live benchmark. The exercise shows a repeatable review process without pretending that one example establishes a model ranking.

Step one gives every model identical evidence

Start with a change small enough to inspect completely. In our example, a customer list supports filtering by account status. The first page respects the selected status, but moving to the next page drops that selection and displays an unfiltered list.

The patch carries the selected status into subsequent page requests. It also adds a regression test confirming that the second page remains filtered. Nothing in the supplied evidence establishes a security issue, a performance improvement, or a change to the filtering interface.

Give all four models the same patch, that short problem description, and the same instruction: write a concise imperative subject and an optional body explaining the behavior and purpose. Require every claim to be supported by the supplied material. Keep repository conventions identical across requests.

Step two keeps routing separate from editorial judgment

BestRelay publishes this exercise as a quality-first API relay that forwards each request to the official model named in it, without substitution. That makes explicit model selection a useful part of the experiment record. Keep candidate identities hidden from the reviewer until scoring is complete.

The same workflow can run through Claude Code, Codex, Gemini CLI, or an OpenAI SDK. Record the requested model, generation settings, and input alongside each response. Identical inputs improve comparability, although one response per model still provides limited evidence.

Track expense separately from writing quality. BestRelay uses pay-as-you-go billing based on provider-reported token counts, with cache reads and writes priced separately. Members can also earn referral commissions when invited users top up; those incentives have no place in the scoring rubric. Create a BestRelay account to try the exercise with your own patch.

Step three compares four messages against the patch

Candidate A proposes: “Update customer pagination.” The reviewer accepts its basic accuracy but finds it too broad. It names the area being changed without identifying the broken behavior. A future search for filtering regressions could easily miss it.

Candidate B proposes: “Preserve account status filters across customer pages.” Its body explains that subsequent page requests previously omitted the selected status and that the regression test covers the second page. The reviewer finds a clear connection between the implementation and the user-visible result.

Candidate C proposes: “Prevent unauthorized customer records from appearing during pagination.” The wording sounds consequential, but the evidence describes filtering, not authorization. The reviewer rejects the unsupported security claim. An ordinary display bug should not acquire a security interpretation through confident phrasing.

Candidate D proposes: “Pass the selected status into subsequent customer page requests.” This accurately describes the mechanism. The reviewer considers it useful but less complete than B because it leaves the reader to infer the filtering failure that motivated the change.

Step four scores claims before polishing sentences

Evaluate each message for factual accuracy, coverage of the behavioral change, usefulness to a future reader, and concision. Treat invented claims as a serious defect rather than letting elegant language compensate for them. Under that rubric, B wins this illustrative review, D follows, A needs detail, and C needs correction.

The accepted message can remain close to B. Its subject identifies the preserved behavior, while its body explains the previous failure and the regression coverage. Avoid adding assurances that the patch guarantees all pagination behavior unless the evidence supports that scope.

Step five tests whether the judgment generalizes

Repeat the exercise across different changes, including refactors, reversions, dependency updates, and patches with ambiguous intent. Collect multiple responses from each model and use another reviewer to expose disagreements. Preserve the original candidates alongside accepted edits. That record reveals whether a model consistently supplies useful intent, merely produces attractive sentences, or needs substantial human correction.