Reliable AI Features Begin With Explicit Failure Rules

A prototype proves that an AI feature can produce a useful answer. Production requires evidence that it stays useful when inputs are messy, dependencies stall, and users discover unexpected behavior. The essential shift is to define acceptable failure as carefully as successful output. Consider a support assistant that drafts replies for human approval: its production journey begins with boundaries, not a more elaborate prompt.

Step one defines the feature's operating boundaries

Write a short contract describing the assistant's inputs, permitted outputs, and authority. It may summarize a ticket and suggest a response, but refund approval requires a separate authorized action. Treat retrieved documents and customer messages as untrusted material, including text that tells the assistant to ignore its instructions.

Define a safe response for missing evidence, unsupported requests, and unavailable dependencies. For this assistant, an explicit handoff to a person is a valid outcome. A confident answer without supporting information is a failure, even when it sounds helpful.

Step two turns examples into release evidence

Build an evaluation set from representative, appropriately redacted tickets. Include straightforward questions, ambiguous requests, conflicting policies, long conversations, and malicious instructions embedded in quoted text. Reserve a held-out portion so prompt revisions cannot simply overfit familiar examples.

Score factual support, policy compliance, useful abstention, and response time separately. Combine automated checks with human review where judgment matters. Establish release thresholds before tuning. A higher average score should not conceal a new failure involving sensitive information or unauthorized promises.

Step three makes model selection an explicit dependency

Record the requested model, prompt revision, retrieval settings, and generation parameters with each evaluation run. Repeat critical tests whenever any of them changes. Provider updates can alter behavior even when your application code remains untouched, so schedule recurring checks.

BestRelay forwards each request to the official model named in it without substitution, across providers including Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, and Grok. That routing property helps keep model selection explicit; application teams still need to verify output quality. Its compatibility with Claude Code, Codex, Gemini CLI, and any OpenAI SDK supports existing development workflows, while production readiness depends on the application's own controls.

Step four contains failures before they spread

Set an overall deadline for each user operation, then allocate time to retrieval, generation, and validation. Retry only eligible transient failures, with bounded attempts and backoff. Retries must fit inside the original deadline. Otherwise, a temporary slowdown can multiply traffic and exhaust capacity.

Keep actions that change external state behind authorization checks and idempotency controls. If a request times out after an action might have succeeded, reconcile its status before repeating it. Validate generated structures and business rules before accepting results. A syntactically valid answer can still propose an impossible refund or cite a nonexistent policy.

Step five connects telemetry to quality and spending

Capture latency, errors, retry counts, validation failures, human corrections, and provider-reported token usage. Use request identifiers to connect events while minimizing retained customer content. Restrict access to diagnostic records and define retention periods before collecting production traffic.

BestRelay bills usage on a pay-as-you-go basis using provider-reported token counts, with cache reads and writes priced separately. Track those categories independently and measure cost per accepted draft, including failed attempts. Keep referral earnings separate from operating costs: members earn a commission when people they invite top up, but that income does not demonstrate application efficiency.

Step six exposes traffic in controlled increments

Start with offline replay, then run the assistant in shadow mode without showing its drafts to customers. Move to a small group of internal reviewers before exposing a limited share of eligible traffic. At each stage, compare quality, latency, and cost against the established baseline.

Define stop conditions and a rollback owner in advance. Preserve a non-AI path, such as the existing ticket queue, so disabling generation does not disable support. Create a BestRelay account to test this workflow with a bounded evaluation budget.

Step seven makes maintenance a continuing release discipline

Review production failures regularly and add representative cases to the evaluation suite. Recheck permissions, dependencies, and provider behavior as the feature evolves. Each release should carry evidence that its intended improvement survives realistic inputs and that its failure boundaries still hold.