Useful AI Monitoring Separates Waiting From Failure

A successful response can still disappoint users

An AI request can return successfully and still feel broken. A long pause before the first word may make someone abandon a chat. A response that starts quickly but stops halfway creates a different problem. Monitoring should reveal both experiences, rather than reducing every request to a single success label.

Start with three questions: Did the request finish, how long did the person wait, and was the result usable? These questions separate technical availability from user experience. A valid response can still contain an unhelpful answer, so application feedback belongs beside your network measurements, not inside the same error count.

Two clocks reveal different kinds of waiting

Total latency is the time from sending a request until receiving the complete response. For streaming requests, also measure time to first token: the wait before generated text starts arriving. Define your measurement carefully, since an initial connection event or empty stream message is not necessarily useful output.

A quick start followed by slow generation differs from a long initial silence. Record both timings, together with whether streaming was enabled. Keep input and output token counts nearby: longer prompts and longer answers can affect duration. Compare similar workloads before deciding that a model or service has become slower.

Averages alone hide frustrating waits. The median describes the middle request; the 95th percentile marks a duration that roughly 95 percent of measured requests meet or beat. Show request volume alongside these values. A percentile based on a handful of calls is much less reliable than one drawn from steady traffic.

Error categories should point toward different actions

Count errors by cause, not just by HTTP status. Authentication failures usually need a credential or permission fix. Invalid requests need changes to the request. Rate limits call for traffic control, while provider failures may justify a later retry. A timeout means your application stopped waiting; it does not prove that generation never happened.

Streaming requires special attention. A request may establish a successful connection and then fail before completion. Record whether the stream reached its expected ending, whether the user cancelled, and whether your application received a usable result. Treat cancellations separately from service failures so ordinary user behavior does not inflate incident counts.

Retries can hide failures while increasing costs

Track each attempt and the overall user operation. If the first attempt fails and the second succeeds, the user sees one successful operation, but your system experienced a failure and additional waiting. Measuring only the final outcome hides that friction.

Retry only failures that may be temporary, with limited attempts and increasing delays. Honor provider retry guidance when available. Avoid automatically retrying malformed requests or invalid credentials. Remember that a timed-out attempt may already have consumed tokens, so retries can increase spending even when they improve completion rates.

Relay measurements need a clearly defined boundary

With a relay, application timing includes the route to the relay, relay processing, the upstream provider, and the return journey. One client-side stopwatch cannot isolate those components. Keep request identifiers and timestamps so support teams can investigate a slow call without guessing.

BestRelay, which publishes this article, is a quality-first API relay that forwards each request to the official model named in it without substitution. It covers Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok, and more, and works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. Record the requested model and client tool to make comparisons meaningful. Create a BestRelay account to try these measurements.

A small dashboard should connect reliability and spending

Begin with request volume, completion rate, latency percentiles, first-token timing, and errors by category. Add token usage and retry counts. Avoid logging full prompts or responses by default; request metadata often provides enough diagnostic value without exposing sensitive content.

BestRelay bills pay-as-you-go using provider-reported token counts, with cache reads and writes priced separately. Keep those usage categories distinct when reviewing costs. Members also earn referral commission when invited people top up; keep that account activity separate from API reliability metrics. Set alerts around sustained user impact, then investigate the affected requests rather than reacting to every isolated spike.