Research: a vector for every article of every federal law — Qwen3-Embedding 0.6B vs 4B, storage and the two-machine workflow (phase 0)

[mgraffg]

Why

We can now walk every federal law's current text down to its structure:
iter_current_federal_laws (issue #191) yields one snapshot per law,
md2akn.parse_markdown turns it into a tree, and every article, fracción and
inciso in it carries an eId. Nothing downstream (retrieval, near-duplicate
detection, clustering by topic, cross-reference resolution — issue #62,
topic-conditioned evaluation over materia — issue #215) can start until each
piece of that text is a vector in some space of dimension K.

This issue is phase zero: fix what gets embedded, run the two models
chosen for the job, and settle how the result is stored and traced back to
the law and article that produced it — and design the workflow across the two
machines involved. It does not implement anything in the packages. Its output
is a written decision, and then a separate implementation-plan issue.

Related: #50 (a text model over the whole DOF archive — this issue is about the
segmented current text of federal laws, a much smaller and structured corpus),
#62 (cross references, a natural consumer), #215 (materia/vigencia per law,
the free labels used below).

Decisions

Three things are decided up front, so the phase measures how well they work
rather than whether:

1. The unit is the article. An article (akn_type == "article": its
   chapeau, its fracciones and incisos, its tail) is the unit a lawyer cites
   and reasons with, and it is the size an embedding model was trained on:
   median 425 characters, 92 % under 2 000 (≈ 500 tokens in Spanish). A leaf
   alone (a fracción's own text, median 165 characters, 12 % under 40 —
   II.- Las personas físicas;) says nothing without its article.
2. Only the long tail is split. An article over 2 000 characters is
   divided at its direct children — one piece per fracción (or per tail
   paragraph), each piece prefixed with the article's chapeau so it still
   reads as the sentence it completes. Articles at or under the cap are
   embedded whole, never split.
3. Two models, one family: Qwen/Qwen3-Embedding-0.6B and
   Qwen/Qwen3-Embedding-4B (Apache-2.0, 32k context, Matryoshka —
   the output dimension is user-set up to 1 024 and 2 560 respectively).
   Same training recipe, same instruction interface, same tokenizer: running
   both on identical inputs isolates exactly one question, how much does
   model size buy on legal Spanish, with nothing else varying.

The text, measured

Run over the whole cached scjn-leyes release on 2026-09-06 (CPU, ~4 min),
walking the tree the way analisis/md2akn_leaf_walk.py does (a local
scaffolding script, not committed).

The corpus: 315 laws, 45.5 M characters of leaf text (≈ 12–13 M tokens for a
multilingual tokenizer), 53 632 articles.

Unit	n	median chars	mean	p95	p99	max
leaf (fracción / inciso / chapeau alone)	211 258	165	215	578	890	12 235
whole article	53 632	425	829	2 749	6 545	247 148
article ≤ 2 000 whole, else split by fracción with chapeau (the unit)	103 169	405	530	1 364	1 919	26 082

Whole articles by size cap:

cap	articles that fit
1 000 chars	78.6 %
1 500 chars	87.4 %
2 000 chars	91.8 %
3 000 chars	95.7 %

The split rule touches 4 418 articles (8.2 %) and turns them into 49 537
pieces — long articles are long because they have many fracciones (about 11
each). 664 pieces are still over 2 000 characters after one split (a single
fracción that is itself a page long); Qwen3's 32k window embeds them whole,
so no second split is needed — they are counted and recorded in the manifest,
not truncated. Repeating the chapeau costs 9 M characters (54.6 M total input
against 45.5 M of leaf text), i.e. 20 % more tokens for the GPU and nothing
on disk.

Two consequences:

• The unit count halves and the input stays in the sweet spot. 103 169
  units instead of 211 258 leaves, median 405 characters, 99 % under 1 919 —
  every vector represents one coherent legal statement of 100–500 tokens.
• A leaf is still addressable. Every leaf maps to exactly one unit (the
  article that contains it, or the piece built from its fracción), so the
  eId granularity is kept as a key, even though the vector is the
  unit's.

What has to be settled

1. The exact text of a unit

The rule above leaves details that have to be written down once and never
changed silently, because they are part of what the vector means:

• The chapeau is the article's first content child; a split article with no
  chapeau (a bare list of fracciones) gets pieces with no prefix.
• Whether the article's num and the law's nombre go into the text
  (Artículo 305, Ley Federal del Trabajo. <chapeau> <fracción>) — cheap,
  and it lets the model tell apart the many articles that say the same
  thing in different laws. To be decided by proxy 3 below, not by taste.
• The Qwen3 instruction/prompt used for documents (none, by the model card's
  convention — the instruction goes on the query side) — recorded in the
  manifest either way.
• Whitespace normalization before hashing (dedup) and before embedding are
  the same function.

2. The two models, and what each run records

Model	Params	K (native, truncatable)	Context	Licence	fp16 weights	VRAM (fp16, short inputs)
Qwen/Qwen3-Embedding-0.6B	0.6 B	1 024	32k	Apache-2.0	≈ 1.2 GB	≈ 2 GB
Qwen/Qwen3-Embedding-4B	4 B	2 560	32k	Apache-2.0	≈ 8 GB	≈ 9 GB

Specs as recalled — re-verify on the model cards before the run. The 4B
is the one the small disk feels: fp16 weights are ≈ 8 GB, so it is downloaded
to the largest partition, run, and deleted; 8-bit loading (≈ 4 GB) is the
fallback if even that does not fit, with the loss measured against the fp16
run on a sample before trusting it. Both models embed the 54.6 M characters
in well under an hour on a modest GPU — compute is not the constraint; disk
for the 4B's weights and an honest comparison are.

3. Dimension K and precision

Bytes for the 103 169 units (dedup by text hash saves a little more — a few
thousand identical transitorios):

K	fp16	int8	binary (1 bit/dim)
256	53 MB	26 MB	3 MB
512	106 MB	53 MB	7 MB
1 024 (0.6B native; 4B truncated)	211 MB	106 MB	13 MB
2 560 (4B native)	528 MB	264 MB	33 MB

fp16 costs nothing measurable in cosine ranking; fp32 is not stored. Because
both models are Matryoshka, K is a storage knob to measure, not a
retraining: the 4B at K = 1 024 against the 0.6B at K = 1 024 is a fair
size-only comparison, and 4B at 2 560 vs 1 024 says what the extra
dimensions buy. int8 (per-dimension scale) is the second knob. Even the
worst cell is far under a GitHub release asset's 2 GB limit, so publishing is
not the bottleneck.

4. Storage layout and traceability

An embedding is only useful if it points back to which text of which law at
which reform. An eId is stable within a snapshot but not across
reforms (articles get renumbered, fracciones inserted), so the key has to
include the snapshot:

• units.parquet — one row per unit, produced on the CPU machine: slug,
  fecha_publicacion, archivo, codNota, article_eId, piece (0 for a
  whole article, 1..n for a split one), piece_eId (the fracción's eId
  when split), text, text_sha1 (normalized). ≈ 55 MB of text, ~15 MB
  zstd. This is the only file that crosses to the GPU machine.
• leaves.parquet — one row per leaf: slug, archivo, eId,
  akn_type → the (article_eId, piece) it belongs to. Keeps the leaf
  granularity as a key without embedding leaves.
• vectors-<model>-<K>.npy — one fp16 (or int8) matrix, one row per
  distinct text_sha1, memory-mappable, no dependency beyond numpy. Text is
  never stored twice: it is reproducible from (slug, archivo, article_eId, piece) and the release.
• rows-<model>.parquet — row → text_sha1, so units ⋈ rows is the
  full trace.
• manifest-<model>.json — model id and revision (commit hash),
  dtype the weights were loaded in, pooling, normalization, the document
  prompt (or its absence), max_seq_length, how many inputs exceeded 2 000
  characters and how many tokens the longest had, K, output dtype and
  quantization parameters, the scjn-leyes release date and
  indice-global.json.gz hash it was computed from, the split cap, and the
  exact unit template. Without this the vectors are unreproducible numbers.

text_sha1 also gives incremental updates for free: after a new reform, only
units whose hash is not yet in rows need embedding — the GPU machine never
re-embeds a corpus.

Published like every other dataset here: as a GitHub release asset (a new
release, e.g. scjn-leyes-vectors, or assets on scjn-leyes itself), never
in git, read back through a disk-first reader the way scjn.release does.

5. The workflow across two machines

CPU machine (this one, has the scjn-leyes cache)
  1. walk articles, apply the 2 000-char split  ──►  units.parquet + leaves.parquet
                                                       │  scp / release draft (~15 MB)
GPU machine (little disk)                              ▼
  2. for each of the two models, one at a time:
       download weights to HF_HOME on the largest partition (4B: ≈ 8 GB)
       dedupe by text_sha1, sort by length, batch, encode
       write vectors-<model>-<K>.npy (+ the truncated K variants, sliced,
             not recomputed) + rows-<model>.parquet + manifest-<model>.json
       delete the weights before the next model
                                                       │  (50–530 MB per model)
CPU machine                                            ▼
  3. proxy evaluation (brute-force cosine over 103k × 2 560 fp16 is fine in
     numpy chunks; no index needed)
  4. pick model, K, dtype, unit template → publish one asset set

The GPU machine needs only units.parquet, sentence-transformers/
transformers and the model of the moment: no scjn, no nota2md, no
corpus cache. That is what makes the small disk workable.

6. How to evaluate with no labelled data

There is no Mexican-federal-law retrieval benchmark. But the release itself
carries labels that cost nothing:

1. Reform pairs (the strongest free signal). The scjn-leyes release
   holds every reform of every law. For an article reformed between two
   consecutive snapshots, its pre-reform text is a near-paraphrase of the
   post-reform one; with all units of the corpus as distractors, the
   pre-reform version should be the 1-nearest neighbour. Recall@1/@10 with
   zero annotation, on real legal edits.
2. materia (issue Add materia, vigencia and resumen to every federal law of the scjn-leyes corpus (#203, items 1/2/5) #215). Leave-one-law-out: predict a law's materia
   from the majority materia of its units' nearest neighbours in other
   laws. Coarse, but it measures topic geometry across laws, not within one.
3. Split pieces of one article. The pieces of a split article should be
   each other's nearest neighbours more than random units are — the check
   that the repeated chapeau does its job, and the measurement that decides
   whether num/nombre go into the template (section 1).
4. Near-duplicates. Identical transitorios must be identical vectors
   (trivial); their near-variants ((sic), punctuation, one changed date)
   should sit above 0.95 cosine — and unrelated units must not.
5. A small hand-written query set. ~30 questions in plain Spanish with
   the article known to answer them (¿quién puede trabajar desde casa? →
   LFT art. 330-A). The only truly downstream check; an afternoon to write.
6. Cost. Units/s, peak VRAM, weights on disk, output on disk — recorded
   in the manifest of every run. This is where 0.6B vs 4B is decided if the
   quality proxies come out close.

Results are results, so they belong on website/ (a findings page), with
the method on this issue.

Deliverables of this phase

• Verify both models' cards: licence, K, context, pooling, document
  prompt convention, weight sizes.
• Write the exact unit template (section 1) and the split rule as a
  function; confirm the 103 169 count reproduces.
• Produce units.parquet and leaves.parquet with their schemas;
  confirm size on the wire.
• Run both models on the GPU machine; record every manifest.
• Run proxies 1–6 for 0.6B @ 1 024, 4B @ 1 024 and 4B @ 2 560 (plus
  int8 and lower-K slices of the winner); write the comparison up as a
  website/ page.
• Decide model, K, dtype/quantization, unit template.
• Fix the storage layout and manifest schema (section 4) as the contract
  the implementation issue builds against.
• Open the implementation-plan issue (where the code lives — a new
  package, scjn, or scripts/ — is decided there, not here).

Out of scope

• No implementation in the packages, no CLI, no release yet.
• No fine-tuning: this phase runs the two models as published. Fine-tuning
  on reform pairs is a plausible later issue and the proxies above would be
  its training signal — worth keeping in mind, not doing.
• No vector index or search service; brute force is enough for 100k rows.
• Only the current text of federal laws. The design must not preclude
  embedding every snapshot later (the full reform history is the same
  release), which is another reason the key carries fecha_publicacion/archivo.