Repository navigation
feat: add top_k cutoff to document retrieval evaluatorsFeat/document evaluators top k - #13795
Open
HareeshDaxton wants to merge 13 commits into
Open
HareeshDaxton wants to merge 13 commits into
HareeshDaxton wants to merge 13 commits into
Conversation
HareeshDaxton
requested review from
davidsbatista
and removed request for
a team
November 17, 2026 17:29
Contributor
|
@HareeshDaxton is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
top_kcutoff to the document retrieval evaluators (Recall@k, MRR@k, NDCG@k, MAP@k) #13794Proposed Changes:
DocumentRecallEvaluator,DocumentMRREvaluator,DocumentMAPEvaluatorandDocumentNDCGEvaluatoralwaysscored the full list of retrieved documents, so getting Recall@1/@5/@10 meant re-running retrieval for every
cutoff or slicing the lists by hand. Slicing by hand also gives a wrong NDCG@k, because the ideal DCG still
counts every relevant document.
top_k: int | None = Noneon all 4 evaluators. When set, only the firsttop_kretrieved documents of each question are scored.DocumentNDCGEvaluatoralso cuts the ideal DCG attop_k(standard NDCG@k), so a perfect top-k ranking scores 1.0.DocumentMAPEvaluatorkeeps dividing by the number of relevant documents (liketrec_eval'smap_cut), andDocumentRecallEvaluatorinmulti_hitmode keeps dividing by all ground truth documents (standard Recall@k).top_kmust be a positive integer, otherwise__init__raises aValueError.top_kis serialized into_dict(). Pipelines saved before this change still load withtop_k=None.None, keeps the old behavior.How did you test it?
0,-1,2.5,True,"3"), serialization round trips, and old dicts withouttop_kmulti_hitand MAPtop_k=2, while slicing by hand gives 0.77calculate_dcg/calculate_idcgwithtop_ktop_klarger than the retrieved list gives the same result as no cutoff, and input lists are not modifiedto_dicttests updated to include"top_k": None.hatch run test:uniton the 4 test files (175 passed), ruff and mypy all pass locally.Notes for the reviewer
The code change is small (one parameter and one slice per evaluator, plus the IDCG cut for NDCG). Most of the
diff is tests. Validation is repeated in each evaluator, the same way
_get_comparison_valueis, to keep eachcomponent self-contained.
This PR was developed with the help of an AI assistant. I have reviewed and understand the changes and ran the relevant tests locally.
Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.