Careful Evidence Selection Beats Sending Every Document

Document answers need evidence, not entire libraries

A document question-answering system has a straightforward job: find relevant information in a collection and use it to answer a question. Sending every document to a model may seem safest, but it spends tokens on material the answer does not need. It can also bury useful details among unrelated paragraphs.

Token-efficient retrieval means selecting enough evidence to answer accurately without carrying unnecessary text. A token is a unit of text processed by a model, often a word fragment. Fewer input tokens can reduce generation costs and processing time, although retrieval itself adds work. The goal is better evidence per token, not simply the shortest possible prompt.

Retrieval separates finding evidence from writing answers

This approach is often called retrieval-augmented generation, or RAG. First, a search system finds promising passages. Then, the language model receives the question and selected passages and writes an answer grounded in that material.

Imagine an employee asking about reimbursement deadlines. The system should retrieve the relevant policy section, its effective date, and any applicable exception. It should not send the entire employee handbook merely because the handbook contains the answer. However, retrieving only a sentence about deadlines could miss a nearby exception for international travel.

Useful chunks preserve boundaries and essential context

Documents usually need to be divided into smaller pieces called chunks. Good chunks follow meaningful boundaries, such as a section, procedure, or group of related paragraphs. Arbitrary cuts can separate a rule from its conditions or a table from its explanatory notes.

Keep useful labels with each chunk: document title, section heading, version, and access permissions. These labels help search narrow its scope and help the model interpret evidence. A little overlap between neighboring chunks can preserve continuity, but excessive overlap wastes tokens by repeating the same material.

There is no universal chunk size. Start with coherent sections and test real questions. Expand a retrieved passage when the surrounding text is necessary rather than making every chunk large in advance.

Search broadly before choosing a smaller evidence set

Keyword search handles exact names, identifiers, and specialized terminology well. Semantic search finds passages with related meaning even when their wording differs from the question. Combining them can improve coverage, particularly when users describe a policy differently from its authors.

After collecting candidates, a reranker can score their relevance more carefully. Deduplication removes repeated passages. Filters can exclude outdated versions or restrict results to the correct product, department, or date. Enforce access permissions before evidence reaches the answer-writing model.

Set an evidence budget, but treat it as a constraint rather than a target to fill. A narrow question may need two passages. A comparison across several policies may need more. Never discard a critical exception just to meet an arbitrary token limit.

Compression must retain the details that change answers

Extracting relevant sentences can reduce input further, provided the result preserves dates, units, qualifications, and references to source locations. Generated summaries can accidentally remove caveats or introduce errors. Keep original passages available for verification and citation.

Measure success with representative questions and checked answers. Track evidence coverage, unsupported claims, input tokens, retrieval time, and total cost. Include questions the collection cannot answer. A trustworthy system should acknowledge missing evidence instead of turning a weak match into a confident response.

Provider billing makes careful token accounting important

BestRelay publishes this guide as a quality-first API relay that forwards each request to the official model named in it, without substitution. Its coverage includes Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok, and more. It works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK.

Billing is pay-as-you-go using provider-reported token counts, with cache reads and writes priced separately. Members also earn referral commissions when invited users top up. Create a BestRelay account to try your document Q&A workflow.

Keep those billing categories separate when evaluating retrieval changes. Cached input is not free input, and a smaller prompt does not guarantee a cheaper overall pipeline. Compare complete costs alongside answer quality: savings only count when the evidence still supports the answer.