feat(nlp): #805 tokenizer rollout — embedding fix + tokenizers/tokenizer_id for all NLP models

[shujaatTracebloc]

Brings every NLP model in the zoo into compliance with the federated tokenizer contract (#805): a run never starts without a tokenizer, so non-HF models must ship a tokenizer.json and HuggingFace models must declare a tokenizer_id.

1. Fix simple_text.py (the original bug)

SimpleTextClassifier had no nn.Embedding — forward() fed Long token IDs straight into nn.Linear, so upload smoke-training failed with mat1 and mat2 must have the same dtype, but got Long and Float. Added a real nn.Embedding(30522, embed_dim) front end + attention-mask-aware mean-pool + the existing head; input_ids.long() handles the text handler's float-cast branch. Matches the sibling simple_token / simple_mlm.

2. HuggingFace models → tokenizer_id (17 files)

Added module-level tokenizer_id (= model_id, the matching tokenizer repo) to every HF model that exposes .config:

• text_classification (9): bert_base_uncased(+_scratch), distilbert(+_scratch), electra, gemma_2, gte_modernbert, modernbert, phi_3_mini
• token_classification (7): bert_token, deberta_v3_token, distil_token, electra_token, modernbert_token, roberta_token, xlm_roberta_token
• masked_language_modeling (1): wide_mini_mlm (subclasses BertForMaskedLM)

3. Custom (non-HF) models → tokenizer.json (real bert-base-uncased)

All non-HF NLP models use vocab_size=30522, so they ship the real bert-base-uncased tokenizer (special tokens [PAD]/[CLS]/[SEP]/[UNK]/[MASK], max id 30521 < embedding table):

• masked_language_modeling/pytorch/tokenizer.json — shared, auto-detected. The whole dir is bert-vocab: 11 scratch nn.Modules (incl. netmedgpt_style_warmstart, an HF wrapper that does not expose .config, so the SDK treats it non-HF) plus HF wide_mini (also bert). One sibling is correct for every model here and clobbers nothing.
• simple_text_tokenizer.json / simple_token_tokenizer.json — distinct names, not auto-detected. Their dirs also contain HF models with different tokenizers (roberta, xlm-roberta, deberta-v3, modernbert, gemma, phi…), and the SDK auto-attaches any sibling tokenizer.json to every model in the dir — which would silently override those models' tokenizer_id. Distinct naming avoids that; upload explicitly: user.upload_model("simple_text", tokenizer="simple_text_tokenizer.json").

Documents the convention in CLAUDE.md.

Verification (feature/748 SDK branch)

• All 17 HF files parse a tokenizer_id.
• All 13 non-HF models are classified non-HF, rejected without a tokenizer, accepted with the shipped one; special-token + embedding-fit checks pass.
• netmedgpt_style_warmstart wrapper confirmed non-HF (.config absent) and satisfied by the shared tokenizer.json.
• Full forward → loss → backward → optimizer.step() smoke-training passes for simple_mlm, simple_token, and simple_text through the real upload handlers.

Targets develop per repo convention.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches every NLP upload path and tokenizer distribution to clients; misnamed or shared tokenizer.json files could silently override HF tokenizer_id in mixed directories.

Overview
Rolls out issue #805 so federated NLP runs always have a declared tokenizer: HuggingFace zoo models get module-level tokenizer_id, and custom nn.Module models ship a tokenizer.json (with naming rules when they share a directory with HF models).

simple_text is fixed so token IDs go through nn.Embedding, attention-mask-aware mean pooling, and the existing head—resolving upload smoke-training dtype errors when Long IDs hit nn.Linear.

CLAUDE.md documents HF vs custom rules, required special tokens, and how sibling tokenizer.json auto-detection can override tokenizer_id in mixed directories.

^{Reviewed by Cursor Bugbot for commit 4ff10d9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 20 / 8

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• .github#57 — fix(fr-gate): pass items at or beyond the required stage · author: @aptracebloc · no reviewer assigned
• backend#813 — test(#805): contract test for tokenizer fingerprint storage (Task 2) · author: @shujaatTracebloc · reviewer: @aptracebloc
• backend#815 — chore(deps): bump cryptography from 47.0.0 to 48.0.1 · author: @dependabot · no reviewer assigned
• cli#78 — fix(dataset rm): delete staging files from a uid-65532 pod, not jobs-manager (#259) · author: @LukasWodka · no reviewer assigned
• cli#79 — chore(schema): re-sync vendored ingest.v1.json from data-ingestors master · author: @LukasWodka · no reviewer assigned
• client#261 — feat(installer): fail fast when HOST_DATA_DIR is on a network filesystem · author: @saadqbal · no reviewer assigned
• client#262 — feat(installer,chart): place datasets on a network mount while MySQL stays local · author: @saadqbal · no reviewer assigned
• client-runtime#108 — fix(authz): match ingest table prefixes at a segment boundary (close cross-tenant straddle) · author: @LukasWodka · no reviewer assigned
• client-runtime#114 — fix(jobs): cap training Job backoffLimit to stop crashloops starving the cluster · author: @saadqbal · no reviewer assigned
• client-runtime#115 — feat(ingestion): run the ingestion pod as the host uid for datasets on NFS · author: @saadqbal · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)