Repository navigation
fix: keep combining marks in the default BM25 tokenizer - #13781
Nipuna-Lakruwan wants to merge 2 commits into
Conversation
|
@Nipuna-Lakruwan is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
1AbdullahButt
left a comment
There was a problem hiding this comment.
I pulled the regex-building code out of the PR head and ran it on its own with the store's lowercase+NFC step. The Sinhala, Tamil, ZWJ, lone-ZWJ and decomposed café cases from the new test all tokenize as expected. Plain Latin/digit text and kana/CJK one-char-per-token are unchanged. I did this on Python 3.10 and 3.12; I didn't run the haystack test suite.
There's one side effect worth deciding on. to_dict() serializes bm25_tokenization_regex as is, so every default InMemoryDocumentStore now writes a pattern of about 4.8 KB into pipeline YAML (4857 chars on 3.12). The pattern also comes from the running interpreter's unicodedata. It differs between 3.10 (Unicode 13.0) and 3.12 (Unicode 15.0), so the same pipeline serializes differently depending on the Python version. A YAML saved on one version also pins that version's mark table. Two options: serialize the default as a sentinel such as None and rebuild it on load, or document this.
One small behaviour change: a combining mark with no base character now becomes a token of its own. "a ́ b" gives ['a', '́', 'b'], while the old regex dropped it. It's probably harmless, but it isn't covered by a test.
Building the class costs about 14 ms at import.
Reviewed with help from Claude Code.
Python's re does not count combining marks as word characters, so the default bm25_tokenization_regex cut Sinhala, Tamil, Devanagari, Bengali and vowelled Arabic or Hebrew words into fragments. Treat combining marks as part of a word, and let a zero-width joiner or non-joiner join two parts of one word, as in Sinhala conjuncts.
The default tokenization regex is about 4.8 KB and is built from the running Python's Unicode data, so writing it into to_dict() made pipeline YAML large and Python-version dependent. Serialize the default as None and rebuild it on load; passing None also selects the default. Also require a word to start with a word character, so a combining mark with no base character is dropped as before instead of becoming a token.
0e967ea to
4fc804e
Compare
|
Thanks @1AbdullahButt, this is a really useful review, and checking on both 3.10 and 3.12 caught the version dependence I hadn't thought through. I've pushed 4fc804e, which addresses both points: Serialization. The default regex is now saved as
One thing to note: YAML saved by earlier versions contains the old default regex as a literal string, so those pipelines keep their saved regex until it's removed or set to Lone combining marks. A word now has to start with a word character, so The branch is rebased on current |
Related Issues
Proposed Changes:
The default
bm25_tokenization_regexofInMemoryDocumentStoreis built on\w, which in Python'sredoes not match combining marks (Unicode categories Mn and Mc). Scripts that write vowels and other signs as marks were therefore cut into fragments: Sinhala "ජනාධිපතිවරයා" becameජන, ධ, පත, වරය, and Tamil, Devanagari, Bengali and vowelled Arabic or Hebrew broke the same way. Those fragments are shared by unrelated words, so BM25 returned documents that don't contain the query word.This PR:
_combining_mark_char_class(), which builds a character class of every combining mark in the Basic Multilingual Plane fromunicodedata. The kana voicing marks (U+3099–U+309A) are left out, so CJK keeps its one-token-per-character behaviour from fix: default BM25 tokenizer splits CJK characters for bare-term retrieval #12837.Latin, digits, CJK and punctuation behave exactly as before. For example, the danda (
।,॥) still separates Hindi words.How did you test it?
test_bm25_tokenization_keeps_combining_marks: Sinhala, Tamil, Bengali, vowelled Arabic, Hindi with dandas, a Sinhala word containing a zero-width joiner, a lone joiner, and decomposed Latin.test_bm25_retrieval_with_combining_mark_script_query: the Tamil query "தமிழ்" returned all three documents before the fix (two of them only shared fragments) and returns only the matching one after it.mainand pass with this change.test/document_stores/test_in_memory.pyandtest/components/retrievers: 728 passed, 19 skipped. The existing CJK tests pass unchanged.ruff checkandruff format --checkpass.Notes for the reviewer
unicodedatarather than hard-code it, because a hand-written list covering all scripts would be about 190 ranges long. Happy to switch to a hard-coded list, or to a smaller set of scripts, if you prefer.to_dict()output when the default is used. If that is a concern, one option is to serializeNonefor the default and rebuild it on load; I kept this PR minimal and didn't change serialization.Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.