WIP .dcf handling

FACIENDA.md

@@ -14,6 +14,22 @@
 - [ ] **CI: tracer pane doesn't open in `dyalog/dyalog` container** — branch `tests-readme`. TestTUI's tracer tests fail in the GH `dyalog` job (and `act -j dyalog`) because Dyalog *itself* declines to send `OpenWindow {debugger:1}` on a breakpoint hit. Captured protocol shows `HadError {error:1001}` + `AppendSessionOutput "B[2]"` (type 5) and `SetPromptType type:1` — then nothing. Local macOS Dyalog at the same version (20.0.52753) sends the `OpenWindow` immediately after. Handshake is identical (`Identify {apiVersion:1, identity:1}`, `Connect {remoteId:2}`); `RIDE_SPAWNED=1` is now set in `uitest.StartDyalog`; `dyalog.dcfg` with `TRACE_ON_ERROR:1` doesn't change behaviour. Adding `DYALOG=<dir>` + `LD_LIBRARY_PATH` (what `session.DyalogEnv` provides) made Dyalog hang silently at startup instead. Diagnosis needs Dyalog-internals knowledge: what does the macOS app provide (`default.dse` overlay? interpreter flag? env var?) that the container image's bare `dyalog +s -q` doesn't, that causes tracer-window emission. Until fixed, the `dyalog` GH job will go red on every push for this reason; the `go` job and most of TestTUI pass.
 - [ ] **harden weak test assertions — false positives whenever rendering is broken** — `uitest.Runner.Test` now gates every predicate on `IsAlive()` (gritt rendering its top border, no `connection refused` / `Press any key to exit` markers). This catches dead-UI runs that previously had dozens of tests trivially passing. But individual predicates are still weak: many use `!runner.Contains("X")`, which is true whenever nothing is rendering OR for any reason X happens not to appear. Each such test needs tightening — assert positive evidence of state-change (e.g. `Contains("X")` BEFORE the action, then `!Contains("X")` AFTER), not just absence at one moment. Audit needed across `tui_test.go`; expected scope ~30 tests.
 
+## dcf
+- [ ] **TUI integration — `:open-dcf` command** — library + grittle (`apldcf`) are in; what's missing is wiring into the TUI. Sketch: `:open-dcf` prompts for a path, calls `dcf.Open`, synthesises a `*codec.Namespace` whose keys are stringified component numbers and values are the parsed bodies, opens a `DataBrowserPane` with a no-op onClose (no Dyalog window). DataBrowserPane's drill-down handles namespaces/arrays/matrices for free. Editing leaves needs writeback in the library first.
+- [ ] **DCF writes (`Append` / `Replace`)** — read-only is shipped. Writes need: per-component checksum (undocumented), journal/backup-region management, header-metadata update on append. RE + synthetic-fixture validation; safety rails (auto-backup, type-narrowing warning, pre/post diff) from §14 of `~/DCF-format.md`.
+- [ ] **DCF v13 layout support** — current parser is v20-clean; older v13 files have a different metadata-header layout within each component block (type_rank lands on the wrong bytes for some real-world v13 files). Needs another RE pass.
+- [ ] **DCF component directory** — bytes 0x28..0x6F of the file header almost certainly contain a component-number → block-offset index. v1 bypasses by scanning for the per-component magic; decoding the directory gives O(1) lookup and is a prerequisite for safe writes.
+
+## codec / amicable / dcf — unify Data conventions
+- [ ] **Decide canonical Data layout for `*codec.Array` and apply consistently** — currently `codec.APLAN` produces nested rows (`Data[i]` is the i-th row slice), `amicable.Unmarshal` produces row-major flat (`Data` is rows×cols elements). Both packages use the same struct but disagree on layout. Workarounds today: `codec.NestRows` helper applied at the amicable→consumer bridge (e.g. `doOpenDCF` in `tui.go`), and `codec.serializeMatrix` defensively handles both layouts via `flattenValue`.
+  - Lossless round-trip guarantees that must be preserved across any refactor:
+    - **amicable**: 220⌶ bytes → Go value → 220⌶ bytes byte-identical.
+    - **dcf**: .dcf bytes (component bodies) → Go value → .dcf bytes; bodies are 220⌶ so they share amicable's guarantee.
+    - **codec → APLAN text** is expected to be lossy (APLAN can't represent ⎕OR opaque blobs, exact type narrowing, locks/stops/monitors).
+  - Audit needed before changing anything: `codec/json.go` (`ToJSON` emits `Data` directly; older JS consumers may rely on nested for convenience), `data_browser.go` (uses `Data[r].([]any)` for matrix row access), `aplanconv` grittle, `mcp/`, `prepl/client.go`, `codec.Get`, all `codec/*_test.go` matrix assertions.
+  - Once decided: amicable produces canonical layout, codec.APLAN converts to canonical, `NestRows` (or whatever helper survives) becomes either unnecessary or specific to "I want JS-friendly nested output". JS-facing JSON shape preserved at the `ToJSON` boundary.
+  - Don't ship in the same change as a feature. Separate branch, separate review.
+
 ## amicable
 - [ ] **decompiler: extend** — multi-line dfns, more system variables, tradfn string literals/locals, embedded function decompilation (different encoding from standalone ⎕OR — see §5.7), nested namespaces
 - [ ] **bytecode synthesis** — generate/modify ⎕OR bytecode in Go, send to Dyalog via `0(220⌶)`

OPERANDA.md

@@ -223,6 +223,20 @@ Go library for Dyalog's `220⌶` binary array serialization format. Named after
 
 **Known limitations:** Multi-line dfns not yet tested. System functions beyond ⎕← and ⎕IO not mapped. Tradfn string literals not yet supported. Nested namespaces not tested. Embedded function members are extracted as opaque `Raw` bytes but can't yet be decompiled standalone — the namespace-embedded encoding differs from standalone `⎕OR` (different literal indices, tradfn-style bytecode structure). See `deliberanda/namespace-unmarshal.md`.
 
+## dcf package + apldcf grittle (new)
+
+Read-only library for Dyalog Component Files (`.dcf`), pure Go, no Dyalog dependency. Sits next to `amicable/` (220⌶) and `codec/` (APLAN).
+
+**API:** `dcf.Open(path)` returns `*File`. `File.Header()` exposes parsed metadata (version, FirstUsed/NextFree, CreatedAt). `File.Components()` returns `[]ComponentInfo` with `{Number, Offset, BlockSize, DR, Rank, Shape}` — types/shapes are peeked without full body decode. `File.Read(n)` deserialises component n's body via `amicable.Unmarshal` (envelope synthesis — the DCF on-disk array format omits the 220⌶ `DF A4` magic but is otherwise compatible, so we wrap and reuse amicable's decoder).
+
+**Format notes:** `adnotata/0013-dcf-outer-format.md`. AA 0E magic at offset 0, version byte at 2 (0x14 = v20), 64-bit pointer-size indicator at 6 (0xA4), FirstUsed u32 at 8, NextFree u32 at 0x10, creation timestamp (Unix seconds) at 0x78. Each component block has a per-file magic `A4 ?? ?? A4` at +4 (auto-detected at parse time — middle bytes vary by Dyalog version). Component scan is brute-force forward sweep + dedup by content fingerprint. v20 files clean; older v13 files have a different per-block metadata layout — degraded support, RE follow-up needed.
+
+**Grittle:** `grittles/apldcf/` — list components or dump them as APLAN. Usage: `apldcf path.dcf` (list), `apldcf -n 3 path.dcf` (dump component 3), `apldcf -all path.dcf` (dump everything).
+
+**Tests:** `dcf/testdata/` holds synthetic fixtures generated via `gritt -l -e "tie ← ⎕FCREATE 0"` etc. — `empty.dcf`, `one_char.dcf`, `one_int.dcf`, `three_components.dcf`, plus J=2/J=3 variants. Safe to commit; no private DCF content involved.
+
+**Pending:** TUI `:open-dcf` integration (sketch in FACIENDA), writes (`Append`/`Replace` — need to RE the per-component checksum and the journal-region management), proper v13 support, decoding the component directory in the file header for O(1) lookup.
+
 ## ibeam package + TUI pane (new)
 
 I-beam (⌶) lookup library at `ibeam/` with TUI pane integration. Two-tier search:

adnotata/0013-dcf-outer-format.md

@@ -0,0 +1,169 @@
+# 0013 — DCF outer container format (reverse-engineered)
+
+Reverse-engineering notes for the on-disk DCF (Dyalog Component File)
+container. Documented purely from synthetic fixtures generated by
+gritt (`dcf/testdata/`). No bytes from private real-world files are
+recorded here.
+
+The 220⌶ wire format (component body) is documented separately in
+`0010-220-ibeam-binary-format.md`. This note is about the container
+that holds those bodies.
+
+Status: enough to read 64-bit, Dyalog-v20, J=1 C=1 files. Writes,
+checksums, journaling levels 2/3, and 32-bit small-span files are out
+of scope here.
+
+## Fixture summary
+
+Generated via `gritt -l -version 20.0 -e "tie ← ... ⎕FCREATE 0"`:
+
+| Fixture                  | Contents                                | Size  |
+|--------------------------|-----------------------------------------|-------|
+| `empty.dcf`              | no components                           |  304  |
+| `one_char.dcf`           | `'hello'`                               | 1104  |
+| `one_int.dcf`            | `⍳5` (narrows to int8)                  | 1104  |
+| `three_components.dcf`   | `'first'`, `⍳10`, `2 3⍴⍳6`              | 1392  |
+| `j2c1.dcf`, `j3c1.dcf`   | `'hello'` with J=2/J=3                  | 1104  |
+
+## File-level header (offset 0)
+
+All multi-byte fields little-endian. Offsets in hex.
+
+| Offset | Size | Meaning                                       |
+|-------:|-----:|-----------------------------------------------|
+|  `00`  |   2  | Magic `AA 0E`                                 |
+|  `02`  |   4  | Dyalog format version (`14 00 00 00` = v20)   |
+|  `06`  |   1  | Pointer-size code: `A4` = 64-bit              |
+|  `07`  |   1  | Flag — 0 if file has never had a component, 1 otherwise (likely `unicode` bit) |
+|  `08`  |   8  | First-used component number (u64; default 1)  |
+|  `10`  |   8  | Next-free component number (u64) — what `⎕FAPPEND` would allocate next |
+|  `18`  |   8  | Offset to start of primary data region — `0` for empty, `0x188` for non-empty (i.e. fixed) |
+|  ...   |      | Several u64 file-size / region-size fields    |
+|  `28`  |   8  | Tail offset / used-bytes marker               |
+|  `30`  |   8  | Same value as `28` (mirror)                   |
+|  `38`  |  12  | Sentinel `FF FF FF FF FF FF FF FF FF FF FF FF` |
+|  `44`  |   4  | Constant `0C 00 00 00`                        |
+|  `70`  |   8  | Constant `F5 01 00 00 00 00 00 00` (= 501; format/build code) |
+|  `78`  |   8  | File creation timestamp (60ths of a second since 1970-01-01 UTC) |
+|  `80`  |   4  | Per-file token (appears verbatim in every component header) |
+
+The header is 256 bytes (0x100). At J=1 the entire file is duplicated:
+header + components are stored once in the primary region, then again
+in a file-backup region near the end of the file. The first byte of
+each block in the backup region is at a 4-byte-aligned offset.
+
+The fields I haven't fully nailed down (`28..67`-ish) include the
+component directory / index. For v1 we scan rather than index — see
+below.
+
+## Per-component block
+
+Each component occupies a block introduced by:
+
+```
+[u32 size_lo]      ← size in bytes of THIS block (primary or journal copy)
+[4 bytes magic]    ← A4 14 0E A4
+```
+
+After the magic, the block contains a metadata header and then a body
+that mirrors the 220⌶ wire format from `0010-`. Key offsets within
+the block (relative to the size_lo field at +0):
+
+| Offset | Size | Meaning                                               |
+|-------:|-----:|-------------------------------------------------------|
+|   0    |   4  | block_size (u32)                                      |
+|   4    |   4  | Magic `A4 14 0E A4`                                   |
+|   8    |   8  | (padding / pointer-size-related)                      |
+|  10    |   8  | Constant `F5 01 00 00 00 00 00 00` (matches file `70`)|
+|  18    |   8  | File creation timestamp (matches file `78`)           |
+|  20    |   4  | Per-file token (matches file `80`)                    |
+|  24    |   4  | Element count / shape product (u32)                   |
+|  28    |   2  | 220⌶ type_rank: low byte `(rank<<4)\|0x0F`, high byte type code |
+|  2A    |   6  | (padding to 8-byte alignment)                         |
+|  30    | rank×8 | shape (u64 each)                                    |
+|  +     |   n  | body (bit-packed / int8 / char8 / float64 / …, padded to 8 bytes) |
+
+For `'hello'` (rank-1 char8, 5 elements):
+- block_size = 0x70 (112)
+- type_rank = `1F 27` (rank=1, simple, type=Char8)
+- shape = `[5]`
+- body = `68 65 6C 6C 6F` + 3 bytes padding
+
+For `⍳10` (rank-1 int8, 10 elements after narrowing):
+- block_size = 0x78 (120)
+- type_rank = `1F 22` (rank=1, simple, type=Int8)
+- shape = `[10]`
+- body = `01 02 03 04 05 06 07 08 09 0A` + 6 bytes padding
+
+For `2 3⍴⍳6` (rank-2 int8, 6 elements):
+- block_size = 0x78 (120)
+- type_rank = `2F 22` (rank=2, simple, type=Int8)
+- shape = `[2, 3]`
+- body = `01 02 03 04 05 06` + 2 bytes padding
+
+## Block layout in the file
+
+Each component appears **at least twice**:
+- Once in the primary region (offset `0x100` and onwards), followed
+  immediately by a journal copy at offset `+0x48` (72 bytes) from the
+  primary.
+- Once more in the file-backup region near the end of the file (the
+  region also contains a second copy of the file header).
+
+`block_size` is the total *content* bytes for that block including
+its header and body, **not** including the inter-copy gap or
+end-of-block padding.
+
+## Reading strategy for v1
+
+Brute-force scan rather than index walk:
+
+1. Open file, verify magic `AA 0E` at offset 0.
+2. Read `next_free` from offset `0x10`; component count = `next_free - 1`.
+3. Scan forward from offset `0x100`, searching for the per-component
+   magic `A4 14 0E A4`. The `u32` at `magic_offset - 4` is `block_size`.
+4. For each unique component, take the **first** occurrence (lowest
+   offset) and parse the metadata header + 220⌶ body.
+5. Deduplicate journal/backup copies by `(timestamp, element_count, body bytes)`
+   — or by ordinal position (every odd occurrence in the primary region
+   is a primary, every even one is its journal, stop before the
+   file-backup region).
+6. Map ordinal → component number using `first_used..next_free-1`.
+
+This is enough for a read-only browser. Writes need the index, the
+checksum, and the journal/backup management — all out of scope here.
+
+## Version-specific layout
+
+The per-component magic's middle two bytes encode the on-disk format
+version: `14 0E` for v20 files, `0D 0A` for older v13-era files. The
+container's outer bytes (`AA 0E` magic, header offsets for
+`first_used`/`next_free`/timestamp) are consistent across both.
+
+What **does** differ between v20 and v13: the position of the
+220⌶ type_rank field within each component block. The v20 layout
+documented above (type_rank at block+0x20) does not hold for v13;
+the v13 metadata header is structured differently. v1 of the parser
+handles v20 cleanly and falls back to a best-effort scan for v13.
+Proper v13 support needs another reverse-engineering pass — out of
+scope for the initial read-only library.
+
+## Open questions
+
+- Exact layout of the component directory / index in the file header
+  (bytes `28..6F` of the header) — currently bypassed by scanning.
+- Encoding of the per-component checksum (the u32 we see after the
+  block in the journal region varies per-component and per-file).
+- How J=2 / J=3 alter the on-disk layout (fixtures `j2c1.dcf` /
+  `j3c1.dcf` show the same size as J=1, suggesting J differs only in
+  fsync/ordering not layout — but unverified).
+- Whether the access matrix lives in the header or in a dedicated
+  component.
+- 32-bit (pointer-size = `94`) files have not been examined.
+
+## Out-of-scope for now
+
+- Component writes (`⎕FAPPEND`, `⎕FREPLACE`) — these need checksum
+  maths and journal-region management. Reads first; writes later.
+- 32-bit small-span files — deprecated, low priority.
+- Access matrix (`⎕FRDAC` / `⎕FSTAC`) byte layout.

codec/aplan.go

@@ -568,6 +568,40 @@ func cellShape(value any) []int {
 	}
 }
 
+// NestRows reshapes an *Array whose Data is row-major flat (the
+// convention amicable.Unmarshal produces for rank ≥ 2) into the
+// nested-rows convention codec.APLAN parses to (Data[i] is the i-th
+// row slice). Other values pass through unchanged.
+//
+// Use at boundaries where flat-Data input must reach a consumer that
+// expects nested rows (e.g. the data browser). Doesn't recurse —
+// nested arrays/namespaces inside Data aren't touched, so callers
+// drilling into compound members may need to NestRows again.
+//
+// Mutates and returns the input Array.
+func NestRows(v any) any {
+	arr, ok := v.(*Array)
+	if !ok || len(arr.Shape) < 2 {
+		return v
+	}
+	rows := arr.Shape[0]
+	cells := 1
+	for _, s := range arr.Shape[1:] {
+		cells *= s
+	}
+	if rows*cells != len(arr.Data) {
+		return v
+	}
+	nested := make([]any, rows)
+	for r := 0; r < rows; r++ {
+		row := make([]any, cells)
+		copy(row, arr.Data[r*cells:(r+1)*cells])
+		nested[r] = row
+	}
+	arr.Data = nested
+	return arr
+}
+
 // flattenValue recursively flattens a value to a 1D slice.
 func flattenValue(value any) []any {
 	switch v := value.(type) {

commands.go

@@ -371,6 +371,10 @@ func buildCommands(cfg *Config) *CommandRegistry {
 		m.loadSession()
 		return *m, nil
 	})
+	reg.add("open-dcf", "Open Dyalog component file (.dcf)", false, "", func(m *Model) (tea.Model, tea.Cmd) {
+		m.openDCF()
+		return *m, nil
+	})
 	reg.add("save-config", "Save current config to disk", false, "", func(m *Model) (tea.Model, tea.Cmd) {
 		m.saveConfig()
 		return *m, nil
@@ -495,6 +499,7 @@ func buildCommands(cfg *Config) *CommandRegistry {
 	reg.alias("cache-refresh", "reload", "update-cache", "redownload")
 	reg.alias("save", "write", "export", "dump")
 	reg.alias("load", "open", "import", "restore")
+	reg.alias("open-dcf", "dcf", "component-file", "ftie")
 	reg.alias("format", "pretty", "indent", "beautify", "prettify")
 	reg.alias("localise", "scope")
 	reg.alias("autolocalise", "auto-scope")

data_browser.go

@@ -325,7 +325,9 @@ func (d *DataBrowserPane) getMatrixCell(m *codec.Array, row, col int) any {
 		}
 		return 0
 	}
-	// 2D+: Data[row] is a []any of columns
+	// 2D+: Data[row] is a []any of columns. The data browser expects
+	// the nested-row convention; callers feeding it a different layout
+	// (e.g. amicable's flat row-major) must reshape at the bridge.
 	if row < len(m.Data) {
 		if rowSlice, ok := m.Data[row].([]any); ok && col < len(rowSlice) {
 			return rowSlice[col]