Good evaluation sets catch failures before users do

Small representative examples beat huge random collections

When people first start judging an AI feature, the temptation is to gather as many test prompts as possible. A large collection feels rigorous, yet size alone tells you very little. Fifty examples drawn from real user requests usually teach you more than five thousand generated guesses, because every one of them carries context you genuinely care about. Begin by reading support tickets, sample conversations, and the documents the feature will actually process. Pick the patterns that repeat, then add the unusual cases that have bitten your team before. A strong first set fits on a screen or two, and every line earns its place.

Treat the set like a sample of the world your feature lives in, not a museum of interesting prompts. If real traffic is mostly short questions about billing, then short questions about billing should appear often, while exotic edge cases stay deliberately rare and clearly marked. When the mix mirrors reality, the score you get at the end means something about the experience your users will have.

BestRelay is a quality-first API relay that forwards each request to the official model named in it, with no substitution between the name you send and the model that answers. That guarantee matters when you measure behaviour, because an unexpected model swap can quietly change every result in your set. The service covers Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok and more, and it works with Claude Code, Codex, Gemini CLI and any OpenAI SDK.

Every example needs one clearly stated expectation

A prompt on its own is only a question. Turn it into an evaluation example by writing down what a good answer contains, which mistakes count as failures, and how much variation you will tolerate. Plain language works perfectly well here; you do not need a formal rubric to be precise. Replace vague hopes such as helpful or professional with concrete markers: the answer must state the deadline, name both required fields, and avoid inventing a policy that is not present in the source material.

Separate your requirements into tiers so disagreements are easy to settle. The first tier holds the facts that must appear, the second holds the tone and format you expect, and the third holds bonuses that are welcome but never decisive. When a reviewer can point at the tier a failure belongs to, scoring stops being a matter of taste. Expectations should also name the boundaries of the task, because a model that politely answers an unrelated question is still wrong.

Keep to one expectation per example wherever you can. Examples that bundle three demands at once make it hard to tell which part failed, and they punish partial success that might otherwise be a useful signal. Narrow examples also survive rewrites better, because a small unit of behaviour is easier to judge consistently across people and across model versions.

Diverse edge cases expose hidden weaknesses reliably

Common cases prove the feature works on a good day. Edge cases prove it will not embarrass you on a bad one. Once the everyday paths are covered, hunt for the inputs that sit at the margins of the problem, and add them in small, deliberate numbers.

• Requests that are unusually short, unusually long, or written in a visible rush.
• Inputs with odd formatting, broken punctuation, duplicated instructions, or stray characters.
• Questions containing conflicting requirements, where one part cannot be satisfied alongside another.
• Prompts that push the feature toward help it should refuse or redirect politely.
• Requests written in a different language or register from the bulk of your traffic.

Each edge case should carry a note about the risk it represents, so future readers understand why it exists. Weight the set so that everyday behaviour dominates the overall score, and report edge-case results separately as well. A single number hides whether a feature is broadly reliable or merely excellent at the average.

Include a few deliberately impossible or underspecified inputs. The correct behaviour there is often to ask for clarification rather than to guess, and that behaviour is easy to lose when a prompt is tuned aggressively for speed and decisiveness.

Running the set repeatedly makes regressions visible

An evaluation set earns its keep through repetition. Run it whenever a prompt changes, a model is upgraded, or a preprocessing step is edited, and keep the scores side by side with the version that produced them. Comparing Claude Opus 5.5 with GPT-5.6 Sol is informative, yet comparing today's prompt against last week's prompt usually matters more, because that is where silent regressions creep in.

Track cost alongside quality, since a prompt that is brilliant but expensive may be a poor trade. Billing at BestRelay is pay-as-you-go from the token counts the provider reports, with cache reads and writes priced separately, which makes those comparisons legible instead of mysterious. A tally of tokens spent per run also tells you when a prompt has grown needlessly long.

Give every run a small amount of human review, even when automated checks pass. Read the failures first, because the interesting signal lives in the examples that changed from correct to wrong. Those cases deserve a permanent place in the set, and they deserve a short note explaining what broke.

Versioned evaluation sets keep teams honest over time

Evaluation sets rot. A feature grows new behaviours, users discover new habits, and last quarter's edge case becomes this quarter's routine request. Treat the set as a maintained artifact: archive old versions, add fresh failures as they appear in production, and retire examples that no longer describe anything real.

Make the rationale behind each example visible to the whole team, and let anyone propose additions with a brief explanation of the risk it guards against. Members earn a referral commission when people they invite top up, which is a small detail of the service's economics, but the larger habit matters more: share the evidence behind quality decisions so debates end in data rather than memory.

Set a review rhythm that matches your release pace, and keep a tiny smoke set that runs in minutes for quick checks. The full set belongs in the pipeline that gates a release, and its results belong in the release notes where stakeholders can see how quality moved.

Create a BestRelay account when you want a consistent place to run these comparisons.