Skip to content

feat: add top_k cutoff to document retrieval evaluatorsFeat/document evaluators top k - #13795

Open
HareeshDaxton wants to merge 13 commits into
deepset-ai:mainfrom
HareeshDaxton:feat/document-evaluators-top-k
Open

HareeshDaxton wants to merge 13 commits into
deepset-ai:mainfrom
HareeshDaxton:feat/document-evaluators-top-k

Conversation

@HareeshDaxton

@HareeshDaxton HareeshDaxton commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

DocumentRecallEvaluator, DocumentMRREvaluator, DocumentMAPEvaluator and DocumentNDCGEvaluator always
scored the full list of retrieved documents, so getting Recall@1/@5/@10 meant re-running retrieval for every
cutoff or slicing the lists by hand. Slicing by hand also gives a wrong NDCG@k, because the ideal DCG still
counts every relevant document.

  • New keyword-only init parameter top_k: int | None = None on all 4 evaluators. When set, only the first
    top_k retrieved documents of each question are scored.
  • DocumentNDCGEvaluator also cuts the ideal DCG at top_k (standard NDCG@k), so a perfect top-k ranking scores 1.0.
  • DocumentMAPEvaluator keeps dividing by the number of relevant documents (like trec_eval's map_cut), and
    DocumentRecallEvaluator in multi_hit mode keeps dividing by all ground truth documents (standard Recall@k).
  • top_k must be a positive integer, otherwise __init__ raises a ValueError.
  • top_k is serialized in to_dict(). Pipelines saved before this change still load with top_k=None.
  • The default, None, keeps the old behavior.
  • Docstrings, the 4 docs pages and a release note are updated.

How did you test it?

  • 64 new unit tests across the 4 test files:
    • defaults, invalid values (0, -1, 2.5, True, "3"), serialization round trips, and old dicts without top_k
    • exact scores at several cutoffs for each metric, including Recall multi_hit and MAP
    • NDCG: a perfect top 2 out of 3 relevant documents scores 1.0 with top_k=2, while slicing by hand gives 0.77
    • NDCG with graded relevance scores, and calculate_dcg / calculate_idcg with top_k
    • top_k larger than the retrieved list gives the same result as no cutoff, and input lists are not modified
  • 7 existing to_dict tests updated to include "top_k": None.
  • hatch run test:unit on the 4 test files (175 passed), ruff and mypy all pass locally.

Notes for the reviewer

The code change is small (one parameter and one slice per evaluator, plus the IDCG cut for NDCG). Most of the
diff is tests. Validation is repeated in each evaluator, the same way _get_comparison_value is, to keep each
component self-contained.

This PR was developed with the help of an AI assistant. I have reviewed and understand the changes and ran the relevant tests locally.

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added unit tests and updated the docstrings
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue (pre-commit is blocked on my Windows machine; I ran ruff and mypy through hatch instead)

@HareeshDaxton
HareeshDaxton requested a review from a team as a code owner November 17, 2026 17:29
@HareeshDaxton
HareeshDaxton requested review from davidsbatista and removed request for a team November 17, 2026 17:29
@vercel

vercel Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

@HareeshDaxton is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added topic:tests type:documentation Improvements on the docs labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/evaluators
  document_map.py
  document_mrr.py
  document_ndcg.py
  document_recall.py
Project Total  

This report was generated by python-coverage-comment-action

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a top_k cutoff to the document retrieval evaluators (Recall@k, MRR@k, NDCG@k, MAP@k)

1 participant