Steady Batch Throughput Starts With Bounded In-Flight Work

Step one separates concurrency from request budgets

A batch job can overwhelm an API even when its average request rate looks reasonable. Starting hundreds of requests simultaneously creates a burst; allowing slow requests to accumulate creates another problem. Predictable throughput requires separate controls for active requests, request frequency, and token consumption.

Begin by recording the limits that apply to your account and chosen model. Check whether they are shared across projects or credentials. A concurrency limit caps unfinished requests, while request and token budgets constrain consumption over time. None of these controls replaces the others.

Step two turns latency into a starting estimate

Estimate your desired requests per second and multiply that figure by typical request duration in seconds. If your target is two requests per second and each request lasts six seconds, roughly twelve active requests can sustain that pace under stable conditions. This is a planning estimate, not permission to exceed documented limits.

Start below that estimate and measure real behavior. Long outputs, large inputs, and variable response times change capacity needs. Leave headroom for interactive traffic and other batch jobs sharing the same allowance.

Step three makes every request acquire permission

Place pending work in a bounded queue and require each worker to acquire an execution slot before sending a request. Release the slot only when the response finishes, fails, or is cancelled. For streaming responses, receiving the first output does not mean the request has finished.

Apply request-rate and token-budget checks separately before dispatch. Estimate token demand conservatively, then reconcile estimates with reported usage when available. Keep producers from filling memory indefinitely: when the queue reaches its capacity, pause ingestion or retain pending items in durable storage.

Step four coordinates limits across every worker

A local semaphore protects one process, not an entire deployment. Ten workers each allowing twelve active requests can collectively send 120. If the upstream allowance is shared, use a shared admission controller or allocate explicit portions of the budget to individual workers.

Keep separate budgets where limits genuinely differ, such as between model families or accounts. BestRelay forwards each request to the official model named in it without substitution, covering Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok, and more. Treat model selection as explicit routing, not an automatic response to congestion.

Step five returns retries through normal admission

Retries must pass through the same concurrency and rate controls as first attempts. Otherwise, a throttling event can trigger a second wave of traffic precisely when the service needs less pressure. Honor retry guidance when supplied; otherwise, use capped exponential backoff with jitter.

Release execution slots during backoff rather than leaving sleeping workers counted as active requests. Bound both attempts and total elapsed time. Retry only failures that are plausibly temporary, and treat ambiguous timeouts carefully: the provider may already have processed the request. Persist item status so restarting a batch does not automatically repeat completed work.

Step six measures throughput alongside spending exposure

Track queue depth, active requests, completion rate, latency percentiles, throttling, retry volume, and tokens consumed. Separate waiting time from service time. A growing queue with stable service latency suggests insufficient admitted capacity; rising latency and throttling suggest that pushing harder may reduce useful throughput.

BestRelay works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. Its pay-as-you-go billing uses provider-reported token counts, with cache reads and writes priced separately. Measure these categories independently when assessing batch cost. Members also earn a referral commission when invited users top up; keep that accounting separate from workload economics. Create a BestRelay account to test a small batch.

Step seven increases concurrency through controlled experiments

Run a representative sample at your conservative starting limit. Increase concurrency in small increments only while successful throughput improves without unacceptable latency, errors, or cost. Reduce it when sustained throttling or deteriorating completion times appear, and allow a cooldown before testing another increase.

Finally, test shutdowns, worker crashes, and delayed responses. Confirm that unfinished items remain recoverable and abandoned permits cannot block progress forever. The goal is not the largest possible worker pool. It is a repeatable operating point that finishes useful work while respecting shared capacity and spending boundaries.