Retrieval evaluation and diagnosis

Why should an unjudged retrieved passage not automatically be treated as definitely irrelevant?

The annotation pool may be incomplete. Review plausible new results or use an evaluation protocol that acknowledges missing judgments instead of penalizing novelty blindly.

A query has four labeled relevant passages and retrieves two of them in its top five. What are recall@5 and precision@5?

Recall is 2/4 = 0.5 and precision is 2/5 = 0.4, under those complete binary relevance labels. Neither directly measures whether the final answer is correct.

Why can reciprocal rank look good for a task that requires evidence from several documents?

It rewards the first relevant result and does not establish coverage of all required facts. Add evidence-coverage or task-success measures for multi-hop questions.

When is nDCG more informative than counting any relevant hit?

When relevance has meaningful grades and rank position matters. Define grades consistently and inspect per-query results; a normalized average can still hide failures on critical slices.

How is approximate-nearest-neighbor recall different from semantic relevance recall?

ANN recall compares approximate results with exact search under the same vector metric. Exact vector neighbors can still be irrelevant to the user's information need.

What must remain fixed when evaluating an ANN search-parameter change?

Embedding model, corpus snapshot, distance metric, query set, filters, and scoring protocol. Compare latency and neighbor recall without mixing in unrelated representation changes.

Why is comparing two chunking strategies at the same top-k not necessarily a fair context-budget comparison?

Chunk lengths and overlap differ. Compare actual tokens and unique evidence delivered, alongside retrieval quality and generation cost.

How can overlapping chunks inflate an apparent retrieval success while reducing useful context?

Several hits may repeat one fact and crowd out complementary evidence. Measure unique evidence coverage and evaluate deduplication or diversification.

Why retain a lexical baseline when testing a dense retriever on enterprise documents?

Exact identifiers, rare terms, and domain vocabulary may favor lexical matching. Measure relevant query slices instead of assuming semantic embeddings dominate every query.

Why is directly adding BM25 scores and cosine similarities often unjustified?

Their scales and distributions differ. Calibrate scores or use a rank-based fusion method, and tune against held-out relevance judgments.

Why can a stronger reranker fail to improve questions whose evidence is absent from the candidate set?

It can reorder candidates but cannot rank an unseen passage into the result. Measure candidate recall separately from ranking quality.

How can applying authorization filters only after top-k retrieval reduce useful recall?

Unauthorized hits consume candidate slots before removal. Use supported filtered retrieval or sufficient candidate generation, while enforcing authorization before any content reaches the model or user.

What should a retrieval evaluation include besides authorized queries?

Cases where relevant but forbidden documents exist, revoked access, and tenant-boundary collisions. Quality gains must not come from exposing inaccessible evidence.

How can you verify that document updates and deletions reach retrieval results?

Use versioned test documents and measure propagation until old content disappears and new content becomes searchable. A successful ingestion job does not prove the serving index changed.

A model fails even with manually supplied sufficient evidence. What should be investigated before improving the retriever?

Instruction interpretation, evidence use, reasoning, output constraints, and the scoring rubric. Retrieval is not the limiting stage for that controlled case.

Can an answer be faithful to retrieved text yet factually wrong?

Yes, when the source itself is incorrect or outdated. Evaluate grounding separately from source authority, freshness, and real-world correctness.

Why is a syntactically valid citation not enough to verify an answer?

The cited passage may not support the attached claim. Check claim-level support and coverage, including unsupported claims that have no citation.

Why include questions that the indexed corpus cannot answer?

They test calibrated abstention and distinguish evidence-based responses from plausible fabrication. Define when missing evidence should trigger a clarification, abstention, or another retrieval step.

Retrieval recall improves but answer accuracy drops. What controlled comparisons can localize the regression?

Hold the generator and prompt fixed, inspect added distractors and token truncation, compare gold-context and retrieved-context runs, and review affected query slices. More retrieved evidence is not automatically more usable evidence.