Failing CI jobs need evidence, not confident agent prose

A failing job hides its cause in layers of context

A red job in a pipeline is a symptom, and the symptom is often several steps removed from the defect. The log stream that accompanies it mixes build output, dependency resolution, test assertions, timing data, and runner health into one long scroll. A coding agent that reads only the final lines will usually find the loudest exception rather than the earliest broken assumption, and the difference between those two readings is the difference between a cosmetic patch and a durable one.

The layers matter. The runner image may have drifted since the last green build, a cached artifact may have been produced by a different toolchain, and a test may pass in isolation yet fail when the suite runs in a different order. Network calls introduce their own uncertainty, since an upstream timeout can masquerade as a logic error. Before any agent proposes a change, the job should be treated as a small system with a history, and that history includes the last successful run.

This is where discipline pays. The engineer who supplies the agent with the failing stage, the first failing assertion, the surrounding timestamps, and the recent change set gives it the context needed to reason. The engineer who pastes the tail of the log and asks for a fix invites a confident guess. Agents are excellent pattern matchers over large text, and they are weakest precisely when the evidence is thin but the tone is certain.

Side-by-side reading separates observed evidence from agent speculation

The most reliable way to audit an agent is to place two accounts of the failure next to each other. On one side sits the agent's explanation: a clean narrative, a named culprit, a proposed fix, and often a sentence claiming the change is safe. On the other side sits the raw record: which stage broke, which assertion failed, which values differed, and which lines of the change set could plausibly touch them. Reading the two together makes the gap between inference and observation visible.

• Evidence first. Every claim in the agent's account should point back to a line, a value, or a timestamp in the record, and claims without a trace belong in the hypothesis column rather than the findings column.
• Mechanism stated. The explanation must describe a causal path from the change to the failure, not merely a correlation such as a shared name or a nearby edit that happened to be touched.
• Scope bounded. The proposed fix should touch the smallest surface that plausibly removes the mechanism, leaving unrelated behavior untouched and keeping the review legible.
• Uncertainty named. A candid report lists what the agent could not verify, such as runner-only behavior, a race that appears only under load, or a dependency that differs between machines.

Once the two readings sit side by side, the conversation with the agent changes character. Instead of defending a story, the agent answers specific questions about supporting observations and falsifying evidence. That turn marks the moment a debugging session stops being a debate about plausibility and becomes an investigation with a direction.

Explicit criteria keep the agent's patch honest

A patch deserves trust only when it survives a checklist agreed before the code is written. The checklist prevents a familiar failure mode: the agent resolves the red job by loosening an assertion, extending a timeout, or retrying a flaky stage, and the pipeline turns green without the defect being understood. Those edits are sometimes legitimate, and they are never acceptable as an unexamined default.

• Reproducible failure. The defect appears under a documented procedure that mirrors the pipeline's runtime, and it disappears when the suspected cause is removed.
• Reversal test. The fix is taken away and the failure returns, which links the change to the outcome instead of to coincidence.
• Minimal diff. The change set contains no opportunistic cleanup, renaming, or dependency bump that would obscure the causal path during review.
• Regression test added. A check captures the mechanism at the lowest level that still exercises it, so a later refactor cannot silently reintroduce it.
• Unrelated signal checked. Neighboring jobs and downstream consumers are consulted before the fix is declared complete, because a green headline can hide a new warning.

When the criteria are explicit, the agent's output becomes easier to reject as well as to accept. Rejection is part of the craft. A model that returns a tidy patch and a tidy story is doing precisely what it was designed to do, and the human remains responsible for the boundary between a plausible story and a demonstrated one.

Reproduction beats reproduction-less reasoning every single time

The strongest lever in any agent-assisted debugging session is a local reproduction. It compresses a long pipeline into a loop of seconds, and it turns vague prompts into precise experiments. Even an imperfect reproduction is valuable, since it narrows the space of possible causes and gives the agent something concrete to argue with instead of a memory of a red icon.

Begin with the smallest surface that still fails: the single test, the single stage, the single input. Then vary one dimension at a time. Pin the runtime, then pin the dependency set, then vary the ordering. The moment the failure vanishes, the variable that changed becomes the suspect, and the agent can be asked to explain that suspect in terms of the code rather than in terms of intuition.

Reproduction also protects the engineer from the most expensive agent failure: a fix that looks correct and passes the pipeline while changing semantics elsewhere. When the same reproduction runs before and after the patch, a claim of safety becomes a measurement. When it cannot run, the honest output is a short list of open questions rather than a declaration of confidence.

The verified fix closes the loop with evidence

Closing the loop means recording what was learned, not only what was changed. A concise report states the failing stage, the observed values, the mechanism, the patch, the tests that guard it, and the remaining risks. That record is the gift the current session leaves the next engineer, and it is the part of the work that survives after the red marker turns green.

The same discipline applies to the tools around the session. BestRelay publishes this article, and it is a quality-first API relay that forwards each request to the official model named in it, with no substitution, across Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok and more. It works with Claude Code, Codex, Gemini CLI and any OpenAI SDK, billing is pay-as-you-go from the token counts the provider reports, cache reads and writes are priced separately, and members earn a referral commission when people they invite top up. Choosing a model is a debugging decision as well, and a patient reader of long logs such as Gemini 3.8 Flash often serves better here than a speed-optimized one.

Create a BestRelay account when you want to run this workflow yourself.