wip(docx): Phase-3 N0 token-map spike — DRAFT, does not compile

[bobzhang]

Warning

DRAFT / WIP — DO NOT MERGE. This branch does not compile. It is a
checkpoint of the Phase-3 N0 token-map spike, parked mid-implementation
when the session switched models. CI is expected red (3 unused-function
errors). Resume plan below.

What N0 is

N0 is the GO/NO-GO gate for Phase 3 (targeted edits). It builds the
projection-to-source token map: for each physical w:p, a map from the
reader's text projection (docx text) back to exact byte spans, precise
enough to cut and splice inside w:t content. Full spec: the # Phase 3
section of docs/docx-agent-roadmap.md (merged in #118), sub-section
"N0 — token-map spike" and "Locked design decisions".

State in this PR (commit ab2aab1)

Done — docx2html/docx/token_map.mbt, data model + walk scaffolding only:

• The five-class inline model as enums: TokenContext
  (RawText/EntityRef/CharRef/CdataText), AtomKind, RestrictionReason,
  BarrierKind; plus ProjToken, ProjAtom, RestrictedSpan,
  BarrierPoint, ParagraphTokenMap, StoryTokenMap.
• ParagraphBuild accumulator; TmFrame / FieldFrame / TokenWalk
  walk-state structs with resolve / inside / innermost_wt accessors.

Not done — resume exactly here:

1. build_story_token_map(part : Bytes) -> StoryTokenMap — the byte-walker
   loop. Model it on scan_story in annotation_scan.mbt (same
   <-dispatch, namespace stack, comment/CDATA/PI skipping, scan_tag
   reuse). This walker is the piece that got overwritten and must be
   re-authored.
2. The w:t content tokenizer: split content into ProjTokens with
   context kinds (raw / & entity / numeric char-ref / CDATA), decoding
   to match the reader (see decode_xml_entities in xml/xml.mbt) while
   recording scalar-only byte boundaries. CDATA-context tokens are
   mutation-refused (v1).
3. The five-class classifier, driven by the reader's ACTUAL behavior —
   docx_reader.mbt is the contract:
   • projecting atoms: w:t, w:tab→\t, w:noBreakHyphen→U+2011,
     w:softHyphen→U+00AD, supported w:sym (see read_symbol);
   • suppressed (bytes exist, no projection): w:del, mc:Choice, move
     wrappers, checkbox-suppressed text (see read_structured_document_tag);
   • restricted (projects, refuses mutation): complex-field cached results,
     w:ins content, non-checkbox w:sdt, mc:Fallback, textbox/pict;
   • transparent seams vs hard barriers per the roadmap table.
4. Story-wide field stack (nested complex fields; fail closed on stray
   end / unclosed-at-EOF / double separate — see read_paragraph_children
   for the reader's per-paragraph version to mirror and extend to
   story-wide).
5. Logical-paragraph enumeration: tracked-deleted paragraph marks merge
   content forward into the next w:p (see read_children +
   read_paragraph_elements_with_prefix); such logical paragraphs are
   TrackedRegion-restricted in v1.
6. The N0 oracle (token_map_test.mbt, new): on the hostile matrix in
   the roadmap — (a) projection equals the reader's, (b) classification
   pins, (c) permitted-pair splice trials byte-identical outside the
   declared edit union with valid re-parse, (d) refusal trials with
   exact declared errors (CDATA / non-scalar / barrier-spanning /
   restricted / suppressed / merged-paragraph / malformed-field).
7. Record the GO/NO-GO verdict in docs/docx-agent-roadmap.md (like L0).

Reviewer note

No review is requested on this draft — it is a resume point, not a
proposal. When implementation is complete it goes through the standard gate
(self-adversarial → subal xhigh → green CI) and this description is replaced.

🤖 Generated with Claude Code