Reliable AI Tests Begin at the Network Boundary

Start by defining behavior your application controls

An AI feature can fail even when the model produces fluent text. Your application might send the wrong conversation, accept an unusable answer, or charge ahead after a timeout. Reliable testing begins by separating those failures from the natural variation in model responses.

Consider a support tool that turns a customer message into a category and a short suggested reply. First, write down its obligations: preserve the customer’s message, request the intended model, validate the category, and route uncertain results to a person. These obligations become testable expectations before you judge the quality of any generated reply.

Place the external call behind one boundary

Next, give your application one component responsible for contacting the AI service. Keep business decisions outside that component. Your support workflow should receive a result or a defined failure without needing to understand network connections or provider response envelopes.

This boundary lets most tests replace the external service with a test double: a controlled substitute that returns a prepared response. Keep the substitute narrow. Recreating an entire SDK produces maintenance work and can hide assumptions that differ from the real client.

Check outgoing requests before judging generated answers

Now test the request your application constructs. Verify the selected model, message roles, relevant conversation history, output constraints, and any enabled tools. Include a case with missing optional information so that accidental assumptions become visible.

Check the contract at the boundary you own. A substitute for your application’s adapter can prove business behavior, but it cannot prove that the SDK serializes requests correctly. Add a smaller set of tests using the real SDK with an intercepted HTTP transport to inspect the outgoing payload without contacting a provider.

Exercise response handling with deliberately awkward fixtures

Build prepared responses that cover more than the happy path. Include a valid answer, missing content, an unexpected category, malformed structured output, and a response stopped by its output limit. If the feature streams text, include interrupted streams and content split across several events.

Assert the resulting application behavior. An invalid category should trigger validation or review. An incomplete response should remain incomplete. Avoid accepting an answer merely because some text arrived, and avoid asserting exact wording when your application only requires a valid category and a usable reply.

Make retries and timeouts observable in tests

Then simulate connection failures, rate limits, and server errors. Inject a controllable clock or delay mechanism so retry tests finish immediately. Verify attempt limits, backoff behavior, and the final error presented to the caller. Account for retries already performed by the SDK so multiple layers do not unexpectedly multiply requests.

Retry policies need special attention when tools cause side effects. Repeating a model request after an ambiguous timeout may create another billable generation. Replaying a tool action may duplicate an external change. Test those decisions separately, using application safeguards such as action identifiers where appropriate.

Use live checks to verify actual integration

After isolated tests pass, run a small live suite with dedicated credentials and a spending limit. Verify authentication, endpoint compatibility, response parsing, and reported usage. Keep these checks separate from the fast suite because network availability and service limits can affect their results.

For applications using BestRelay, include a live check through the relay. BestRelay forwards requests to the official model named in each request without substitution and supports Claude Code, Codex, Gemini CLI, and OpenAI SDKs. Create a BestRelay account to try this walkthrough with your own integration.

Evaluate answer quality with a stable dataset

Finally, maintain representative customer messages with expected categories and explicit reply criteria. Measure category accuracy and review replies for relevance, unsupported claims, and appropriate escalation. Repeat selected cases to expose variation, and compare results when prompts, models, or application logic change.

Track cost alongside quality. BestRelay uses pay-as-you-go billing based on provider-reported token counts, with cache reads and writes priced separately. Record those categories separately when available. Keep any referral commission earned from invited members’ top-ups outside test-cost calculations, so your evaluation budget reflects actual request usage.