Gemini CLI and Claude Code on the Same Task

A shared Python task makes the comparison concrete

Gemini CLI and Claude Code become easier to evaluate when they receive the same repository, instructions, and acceptance checks. This walkthrough gives both tools one bounded assignment: add JSON output to a Python command-line utility without changing its existing text output. It is a procedure you can reproduce, rather than a claim that either tool has already won.

Use a small repository containing a script named report.py. It reads a CSV file with category and amount columns, totals amounts by category, and prints one category per line. The requested change is a --json flag that returns a JSON object containing those totals.

Step one creates matching environments for both tools

Install Node.js at a version supported by both projects, then install the tools with npm install -g @google/gemini-cli and npm install -g @anthropic-ai/claude-code. Check their current official installation documentation for platform requirements. Record the versions returned by gemini --version and claude --version.

Commit your starting repository, then create separate Git worktrees with git worktree add ../trial-gemini -b trial-gemini and git worktree add ../trial-claude -b trial-claude. Both branches should begin at the same commit. Keep dependencies, shell permissions, and available tools equivalent, and check that neither workspace contains extra agent instructions.

Step two separates authentication from the coding comparison

Authenticate each tool using its documented method. If you use a relay, verify the current configuration instructions for each client; endpoint paths and authentication settings can differ. Do not assume that setting one OpenAI-compatible base URL configures every client correctly.

This article is published by BestRelay, a quality-first API relay that forwards each request to the official model named in it, without substitution. It covers Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok, and more, and works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. Its API base is https://api.bestapirelay.com. Create a BestRelay account if you want to follow along through a shared API provider.

Record the selected model and authentication route for each run. Different models introduce another variable, so interpret the results as a comparison of the complete configurations you tested.

Step three gives both agents identical acceptance criteria

Start gemini inside trial-gemini and claude inside trial-claude. Paste the following prompt into each session, unchanged:

Add a --json flag to report.py. Preserve the existing text output when the flag is absent. JSON output must be one object mapping category names to numeric totals, with keys sorted alphabetically. Empty input with valid headers must produce an empty object. Use only the Python standard library. Add regression tests using unittest, run them, and explain the changes. Do not modify unrelated files. Ask before installing dependencies or accessing external services.

Let each agent finish its first attempt before intervening. Record clarification questions, permission requests, commands, and manual corrections. If an answer affects the requirements, provide the same information to the other agent.

Step four checks behavior beyond the generated explanation

Run python -m unittest discover -v in each worktree. Then independently exercise the script, adapting these commands if the existing interface requires different arguments: python report.py sample.csv and python report.py sample.csv --json.

• Use repeated categories to verify aggregation.
• Use decimal amounts to expose numeric handling problems.
• Use a file containing only headers to check empty output.
• Compare ordinary text output against the starting commit.
• Parse JSON output and inspect key order separately.

Review each diff for unnecessary restructuring, changed error handling, and tests that merely repeat implementation assumptions. Passing an agent's own tests is useful evidence, but independent checks establish whether the requested behavior works.

Step five compares verified results with measured costs

Keep a compact record for each run: acceptance checks passed, elapsed time, human interventions, changed files, and billed usage. A faster first response may still require more repair. Likewise, a larger patch may add maintenance work without improving the feature.

BestRelay bills on a pay-as-you-go basis using provider-reported token counts, with cache reads and writes priced separately. Compare those categories separately when available, since warm context can change costs. Members also earn a referral commission when invited users top up; keep that separate from the technical evaluation.

Choose the configuration that delivers correct behavior with an understandable patch and an acceptable amount of supervision. Repeat the exercise on another task before treating one result as a general preference.