Small Concurrency Limits Keep Batch Work Predictable

A small worker pool makes batches easier to control

A batch job turns a collection of inputs into a collection of results. That might mean summarizing support tickets, extracting fields from documents, or reviewing product descriptions. Sending every request immediately feels fast, but it can overwhelm an API, exhaust local memory, and turn temporary errors into repeated failures.

Concurrency is the number of requests in progress at the same time. A concurrency limit puts a ceiling on that number. With a limit of four, your job starts four requests and waits for a slot to become free before starting another. The remaining work stays in a queue.

Concurrency and request rates measure different things

A concurrency limit controls simultaneous work. A rate limit controls activity over a period, such as requests per minute or tokens per minute. These limits are related, but neither replaces the other. Four fast requests can finish and restart often enough to exceed a request allowance.

Long inputs and large responses can also consume a token allowance before you reach a request allowance. A reliable batch runner therefore needs both a cap on active requests and a way to pace new requests. A low concurrency setting alone does not guarantee compliance with provider limits.

A shared queue prevents accidental traffic multiplication

The simplest design has one queue and a fixed number of workers. Each worker takes one item, sends its request, records the outcome, and then takes another item. Keep queued items lightweight rather than loading every document into memory before processing begins.

Count all workers that share the same allowance. Three machines with four workers each can create twelve simultaneous requests. Separate scripts, scheduled jobs, and interactive tools may also compete for the same capacity. Where limits are shared, coordinate access centrally or assign each job an explicit share.

Early measurements should determine your next adjustment

Begin with a small limit and a representative sample, including some longer inputs. Track completed items per minute, request duration, rate-limit responses, retries, and token usage. The goal is useful throughput, not the largest possible number of open connections.

Increase concurrency gradually while results remain stable. If throughput stops improving, latency rises sharply, or rate-limit responses become frequent, step back. Reserve some capacity for unexpected demand rather than treating every published allowance as a target to fill continuously.

Retries must return through the same traffic controls

Retrying immediately can make congestion worse. For temporary failures, wait before trying again and increase that wait after repeated failures. Add a small random variation so many workers do not retry together. Honor retry guidance from the service when it is provided.

A retry is still a request. It should pass through the same concurrency and pacing controls as new work. Set a maximum number of attempts, and avoid retrying permanent problems such as invalid input. A timeout can leave completion uncertain, so use idempotency features when available and account for possible duplicate processing.

Relay access still requires deliberate capacity planning

For readers running batches through BestRelay, the same principles apply. BestRelay is a quality-first API relay that forwards each request to the official model named in it, without substitution. It works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. Those access options do not remove the need to manage traffic across jobs sharing an allowance.

BestRelay uses pay-as-you-go billing based on provider-reported token counts, with cache reads and writes priced separately. Track usage alongside throughput: retries and larger outputs can change costs even when the concurrency limit stays fixed. Members also earn referral commission when invited users top up; keep that separate from workload budgeting. Create a BestRelay account to try a small, measured batch.

Saved progress makes interruptions cheaper to recover

Give each item a stable identifier and save successful results as the job advances. Record failed items separately, including their attempt counts. After an interruption, resume unfinished work instead of rerunning everything. A modest worker pool, sensible pacing, bounded retries, and durable progress are usually a better starting point than a complicated scheduler.