Bring back character q-grams as a token generator

[sadit]

Summary

Adds QgramGenerator(q), a token generator that emits character q-grams. They are meant as encoders for document-vs-document tasks (classification, clustering, RandomIndexing into bit or multi-bit sketches), not for short-query search.

tc = TextConfig(tokenization=TokenizationConfig(nlist=[1],       # words; drop for q-grams only
        generators=[QgramGenerator(3), QgramGenerator(5)]))
voc = filter_tokens(t -> t.ndocs >= 3, Vocabulary(tc, corpus))   # prune the Zipfian tail
model = VectorModel(voc)

• Q-grams are taken over the whole normalized text, boundary blanks included, so a window can span a word boundary. They do not span two fields of a multi-field document.
• Runs of blanks count as one blank, without changing the normalized text that other generators read.
• Tokens are tagged 'q'. The tag keeps the 3-gram que apart from the word que, and keeps word-level stopwords and lemmas from matching q-grams.
• Q-grams use the ordinary Vocabulary, so VectorModel, EntropyWeighting, RandomIndexing, LSI and filter_tokens work unchanged. Because the filter_tokens predicate sees the token, words and q-grams can be pruned with separate thresholds by checking the tag.

Saving

• A profile with q-gram generators is saved as format "1.2". A build that only reads "1.1" refuses it, instead of silently tokenizing without the q-grams.
• Profiles without generators, the published ones included, still write "1.1" with the same bytes and the same profile_id.
• Other custom generators still cannot be saved, and merge_profiles still rejects any config with generators.

Cost

A hash-keyed separate vocabulary was tried first. On 23,186 Markdown paragraphs with 16 threads, both versions built the same space: 314,775 distinct q-grams and the same number of nonzero entries.

	build vocabulary	vectorize	memory
hash-keyed (dropped)	1.2 s	0.32 s	9 MB
QgramGenerator	7.5 s	0.86 s	19 MB

We kept the strings because a Dict is needed either way, and strings can be inspected. Most of the build-time gap comes from Vocabulary taking a lock per token (_locked_tokenize_and_push), which slows the word path equally. That is left for a separate change.

Test plan

• Full test suite passes locally (Pkg.test()), including 33 new tests in test/testqgrams.jl: tokens, tagging, stopwords, pruning, entropy weighting, RandomIndexing, and a save/load round trip in both format versions.
• Docs build (Documenter) not run.

🤖 Generated with Claude Code