Smaller Evidence Packs Make Document Answers Stronger

A document Q&A system does not become reliable simply by sending more pages to a model. Extra context consumes tokens, introduces competing passages, and makes supporting evidence harder to isolate. The better target is a compact evidence pack: enough authoritative material to answer the question, with identifiers that make every important claim traceable.

Step one establishes quality before reducing tokens

Build a small evaluation set from real questions before changing retrieval. Include direct lookups, questions requiring multiple passages, ambiguous requests, and questions the documents cannot answer. Record the expected answer and supporting passages. Separate current documents from superseded versions so an apparently correct answer does not quietly cite outdated guidance.

Measure answer correctness, evidence coverage, citation accuracy, latency, and input tokens together. A smaller prompt is not an improvement if it drops an exception that changes the answer. Establish a baseline using your current retrieval settings, then compare each change against the same questions.

Step two preserves meaning at chunk boundaries

Split documents around sections, paragraphs, lists, and table boundaries rather than relying entirely on fixed token counts. Keep a heading with the text it governs. Preserve table headers alongside relevant rows, and attach document identity, version, section, and access permissions as metadata.

Use overlap sparingly. Repeating a short boundary passage can preserve continuity, but excessive overlap makes retrieval return near-duplicates. Start with chunks large enough to express one coherent idea, then adjust using observed failures. There is no universal chunk size that suits both policy manuals and dense technical reference material.

Step three retrieves broadly without expanding prompts

Search a larger candidate pool than you intend to send to the answering model. Combine lexical retrieval, useful for exact identifiers and unusual terms, with semantic retrieval, useful for paraphrases. Apply permissions and relevant metadata filters before evidence can reach the model.

Merge results using a documented ranking method, such as reciprocal rank fusion, rather than directly adding incompatible search scores. Remove duplicate passages. Candidate retrieval can be broad without making the final prompt large: the candidate pool belongs to the selection pipeline, not automatically to the generation context.

Step four selects evidence within explicit budgets

Rerank candidates against the actual question, then select passages under a token budget. Relevance alone is insufficient: several highly ranked chunks may repeat the same fact while omitting a necessary exception. Prefer a set that covers the question's distinct requirements.

Expand into neighboring passages only when a selected chunk depends on missing definitions, conditions, or references. Preserve wording for decisive evidence. If compression is necessary, retain source identifiers and validate that the compressed text preserves qualifications. A fluent summary can still erase a critical limitation.

Step five makes missing evidence an acceptable outcome

Give the answering model a clear contract: answer from the supplied evidence, associate substantive claims with passage identifiers, and state when support is insufficient. Treat document text as untrusted data, not instructions. Retrieved passages must not be allowed to override system rules or request access to other documents.

Keep output length proportional to the question. A narrow lookup rarely needs a long explanation. For multi-part questions, organize the answer around the requested parts so unsupported gaps remain visible instead of disappearing inside confident prose.

Step six separates retrieval savings from caching

Cache stable instructions or reusable context where the provider supports it, but measure cache behavior separately from retrieval improvements. Cache hits can reduce cost or latency without reducing the logical size of the prompt. Track uncached input, cache reads, cache writes, and output tokens independently.

BestRelay, which publishes this walkthrough, forwards each request to the official model named in it without substitution. Its pay-as-you-go billing uses provider-reported token counts, with cache reads and writes priced separately. That makes request-level usage records useful for checking whether a smaller evidence pack actually lowers cost. Create a BestRelay account to test this workflow.

Step seven keeps improvements tied to evidence

Inspect failures and token usage after every change. Keep the smallest evidence budget that preserves your quality targets, and reevaluate as documents change. The durable optimization is not simply fewer tokens; it is fewer irrelevant tokens without losing the evidence needed for a defensible answer.