A Safe Retry Starts Before the Request Leaves

A timeout leaves the request outcome uncertain

You send an AI API request, wait for an answer, and receive a timeout. Trying again feels obvious. But the timeout tells you only that your application did not receive a response in time. The provider might still be generating an answer, or it might have finished while the connection failed.

A second request can therefore create a second generation, with additional token usage and potentially different text. If the answer drives another action, such as sending an email, the consequences can extend beyond an extra bill.

Idempotency means repeating an operation has the same intended effect as performing it once. For AI requests, an idempotent retry mechanism recognizes repeated attempts as one logical request instead of starting fresh work every time.

One logical request needs one stable identity

A common approach assigns an idempotency key before the first attempt. The client reuses that key when retrying the same operation. A service that supports this mechanism can associate the key with the original request and return its saved result, or report that processing remains underway.

The key identifies the operation, not merely its contents. Two deliberate requests with identical prompts may represent two separate tasks and should normally have different keys. A retry after a dropped connection should keep the original key.

Keep the request unchanged across those attempts, including the model, messages, and generation settings. Changing the prompt while retaining the key creates ambiguity. Some services reject that mismatch; others have different documented behavior. An edited request should receive a new identity.

Support must exist across the request path

Adding a key does nothing unless the receiving service explicitly supports it for that endpoint. Check the documented rules: whether concurrent attempts are combined, whether completed responses are replayed, which failures are retained, and when keys expire. Once a record expires, repeating its key may start new work.

This matters when a relay sits between your application and a provider. BestRelay is a quality-first API relay that forwards each request to the official model named in it, without substitution. It covers Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok, and more, and works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK.

Those compatibility and routing properties do not themselves establish an idempotency guarantee. Confirm the behavior of the particular endpoint and the entire retry path. A record in your application cannot prevent duplicate upstream generation if the first upstream outcome is unknown and you simply submit again.

Retry timing and duplicate prevention solve different problems

Backoff controls timing; idempotency controls repeated effects. For temporary failures, use a limited number of retries with increasing delays and a little randomness, often called jitter. Respect a retry delay supplied by the service. Avoid repeatedly retrying invalid input or failed authentication without fixing the cause.

Also check whether your SDK, relay, or job runner already retries automatically. Several independent retry loops can multiply attempts unexpectedly. Set an overall deadline and record the logical request identity, attempt count, and any provider request identifiers available for troubleshooting.

Streaming and downstream actions need separate safeguards

Streaming makes recovery trickier because the user may already have seen part of an answer. A new generation is not necessarily a continuation of that text. Unless resumable streaming is explicitly supported, do not silently append a retry's output to an interrupted answer.

Protect downstream actions separately, too. Even a replayed response can trigger an email or database update twice if your application processes it twice. Track completion of each business action using a durable operation identifier.

Usage accounting should remain separate from retry assumptions

Prompt caching is not duplicate prevention. BestRelay bills pay-as-you-go using provider-reported token counts, with cache reads and writes priced separately. A cache hit may change cost, but it does not prove that two attempts became one operation. Members also earn a referral commission when people they invite top up; that is separate from request accounting.

Create a BestRelay account to explore the service. Before enabling automatic retries in production, test timeouts, simultaneous duplicates, and interrupted streams. The goal is not endless persistence: it is a clear, bounded recovery process that avoids turning uncertainty into duplicate work.