Corgi-DDIR work#795
Draft
frankmcsherry wants to merge 69 commits into
Draft
Conversation
…, value_id) A boundary where only integers cross: a storage backend presents each record as ((key_hash, value_id), time, diff) — integer proxies for data it keeps in its own layout — the operators own all the lattice/time logic over those integers, and the backend supplies value semantics via callbacks. Any columnar (or otherwise opaque-to-DD) value store can then reuse join and reduce without materializing values. - trace/chunk/int_proxy: ProxyChunk, a cursor-less Chunk of proxy columns, with from_unsorted (integer sort+consolidate with representative provenance) as the presentation-building helper. - operators/int_proxy: ProxyJoinTactic / ProxyReduceTactic for the join_with_tactic and reduce_with_tactic seams (made pub here), and the backend traits: present-as-proxies (read), value callback with hash-minted output ids (write), materialize (egress). Reduce output ids are content hashes, so an output arrangement re-presents with the same ids downstream with no registry; pending interesting times are keyed by the stable key_hash across retires. The module doc carries the boundary contract and design notes (why value_id is not order-preserving; collision risk); each tactic and the in-memory reference backend (VecChunk arrangements, fnv hashes) sit in their own file under the module. - Tests: join and count/distinct/min reduces against the row operators over multi-round retracting inputs, and a scripted Product-time retire sequence exercising synthetic corrections and pending. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e, not a requirement value_id never outlives one computation (a join unit's presentations; one reduce retire, whose materialize resolves ids to real data before anything leaves). The actual contract is a per-computation bijection with value equality, plus within-retire agreement between the output presentation and minted ids. Content hashing discharges all of it statelessly (the reference backend's choice); exact schemes — dense ordinals from grouping, a per-retire value→id map — are equally valid and collision-free. Only key_hash must be a stable pure function of the key (cross-retire pending, changed-key filter), making the key side the irreducible collision exposure. Persisting ids into an output arrangement itself would force stable value ids; this design does not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…worse than the cursor tactics Two regressions found by asking exactly that question, both fixed: - The reference reduce backend implemented the changed-key restriction by scanning every batch and filtering — O(trace) per retire where the cursor tactic seeks, O(delta·log). Presents now seek the changed keys (novel keys resolve from this retire's delta-sized input batches; pending keys from the retire that pended them, which is exactly what the persistent hash→key map retains — pruned to the changed set each retire, so it is bounded by the delta, and the per-retire value map is cleared). - The join interface had no restriction at all, forcing ANY backend to present the entire accumulated side per fresh batch — O(trace·log) per unit where the cursor join seeks. present0/present1 now take an optional sorted key-hash filter; the tactic presents the fresh side first and passes its key set for the accumulated side, the join analogue of reduce's changed-key restriction. The check is mechanical, not wall-clock: counting backend wrappers measure presented records — with 20k arranged keys and five single-key rounds, reduce presents 37 records and join 10, where the scanning versions present Ω(rounds·N) (join: 100,005) and fail the gate. What remains above the cursor tactics is a log-factor sort of delta-sized presentations, and per-(key, wave) rescans of a changed key's presented range where the row replayer consolidates progressively — same worst-case order, different constants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cle fuzz The minimal harness for the framework, closed over itself: the tactics demand only BatchReader of the trace, and ProxyChunk is already a Chunk, so a batch of proxy chunks is the minimal arrangement — the proxy data IS the data, no separate chunk class needed. The identity backend makes values u64s with the identity as the id function: no hashing, no resolution machinery, no collision possibility, and materialize emits the proxy records verbatim (incidentally exercising ChunkBatch<ProxyChunk> as a real output batch). What remains under test is exactly the framework's own contribution — interesting-time discovery, desired-vs-current deltas, pending, held routing — fuzzed over Product-time grids: 300 random inputs retired through random diagonal frontiers (so synthetic joins arise inside and across intervals and must pend), driven through an emulation of the reduce driver protocol, and checked against a brute-force oracle at every grid point: the accumulated output must equal the reduction of the accumulated input, everywhere, for count/distinct/min-shaped reducers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An ignored benchmark (cargo test --release --test int_proxy -- --ignored --nocapture) compares full stacks — the stock row operators against chunk arrangements plus the proxy tactics over the reference backend — for a bulk load and a steady-state incremental phase (warmed past the post-load merge-amortization transient). The bench caught what the counting gates could not: the reference backend's hash→key map was pruned by retain (and the per-retire value map by clear), both of which keep the backing table — so after a million-key load, every retire walked a million-bucket table to visit one entry, ~130µs/round of pure capacity. shrink_to_fit after the prune and a fresh map per retire fix it. Steady-state single-key rounds after the fix, proxy vs row: reduce ~1.4x at every scale (flat from 10k to 1M keys — the delta- proportionality gates hold in wall clock too), join at parity below 1M (~1.0x). Bulk load carries the presentation layer's constant factor (per-record hashing, clones, by-hash sort): reduce 3-5x, join ~2x — the costs a columnar backend's bulk primitives are meant to attack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The longer-term goal for the boundary is fewer crossings into the backend's value logic (interpreted or columnar execution pays per-call overhead): each crossing should carry a list of keys and a longer bracketed list of value entries. ProxyReduceBackend gains reduce_many(keys, ends, input) — group_offsets -shaped brackets, one per key, each non-empty — returning concatenated per-key outputs with their own bracket ends. A default implementation loops the per-key reduce, so simple backends implement only that; backends with bulk value logic override reduce_many. The tactic now calls only reduce_many: retire's key-major loop becomes two passes — derive each changed key's active times, group the work into waves by time, then play the waves in ascending order (Ord extends the partial order, so a key's earlier deltas always precede a later time's reads) with at most one callback per wave, batching every key active at that time. The identity backend overrides reduce_many (asserting the bracket protocol from the backend's seat), so the grid-oracle fuzz exercises the batched path; the reference backend uses the default, so the row-comparison tests cover the loop. All gates and benches unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The desired output at a (key, time) moment is a function of the key's input accumulation at that time alone — no output-side state — so no time ordering constrains the batch: every moment of the retire can share one reduce_many call, a key contributing one bracket per active time. The order-sensitive part (subtracting the current output, which includes deltas emitted at earlier moments) is pure proxy-space arithmetic and moves to a separate pass that plays the moments in ascending time order. The bracket, not the key, is reduce_many's unit: keys may repeat. Contract docs updated accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…linear The row suite's reduce_scaling/join_scaling shapes (one key, many distinct times, one batch) exposed quadratic behavior in both proxy tactics: reduce rescanned a key's full presented range per interesting time (and its interesting-time closure joined all pairs), and join cross-produced full matched histories pairwise. Measured: reduce 6.4s at scale 10k, 4.7x per doubling; join 9.8s at 10k, >90s at 20k. The robust versions are the cursor variants with integers in place of keys and values, as intended: - history.rs: IdHistory, the id-space ValueHistory — (value_id, time, diff) edits replayed in ascending time order into a buffer repeatedly advanced by the meet of the times still to come and consolidated. The advancement is the collapse that keeps a key with many distinct times linear: accumulations read the small buffer, never the raw history. The presentations serve as the fused per-key load. - reduce: discover_and_accumulate ports history_replay::compute — lazy interesting-time discovery (novel and pending seed; synthetics from joins with the advanced batch buffer and times_current) replaces the eager join-closure, and per-moment accumulation reads the advanced buffers. Phase B replays the output side per key over the discovered moments with the same machinery (suffix meets; emitted deltas advanced and consolidated). Still one batched reduce_many crossing per retire. - join: join_key ports JoinThinker::think — each side's edits replayed against the other side's advanced buffer (identical emitted times: t0 ∨ (t1 ∨ meet) = t0 ∨ t1), with the dead-simple cross product kept for small histories. proxy_reduce_scaling / proxy_join_scaling (scale 100k, the row tests' shapes) now pin this; scale 10k dropped from seconds to milliseconds and growth is ~4x per 4x scale. The grid-oracle fuzz over partially ordered times, the scripted pending test, the row comparisons, the delta-proportionality gates, and the steady-state bench all pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The row-vs-proxy comparison on the scaling shapes showed proxy reduce still superlinear (9x per 4x scale; timeout at 4M where row takes 0.5s). Profiling pinned it outside the tactic: the reference backend's materialize built ONE giant VecChunk and let the builder settle it, and settle's split path peels TARGET-sized pieces off the front with split_off, copying the remaining tail each iteration — O(m²/TARGET) in the batch size. Feeding the builder TARGET-sized chunks directly fixes it: 4M drops from >120s to 3.2s. With that, both operators are in the row implementations' complexity class on the scaling shapes (growth ~5x per 4x scale ≈ n·log n; the hot frames are the presentation sort and fnv hashing — constants, not structure): at scale 4M, join row 0.96s / proxy 3.6s, reduce row 0.48s / proxy 3.2s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TimelyDataflow#778 Chunk split) A columnar differential-dataflow backend for DDIR whose native representation is corgi Value columns and whose scalar logic is corgi `eval_graph`, parallel to `backend::vec`. Arrangement: - CorgiChunk : Chunk (NOT NavigableChunk) — corgi columns + Vec time/diff, ordered (key,val) by corgi structural order then time. merge/extract/advance/ settle ported from the VecChunk reference; drives corgi's discrimination sort (sort_perm) + batched compare_idx, never per-pair. Rides TimelyDataflow#778's cursor-less Chunk path, so it gets the fueled/graded ChunkBatchMerger for free. - Tactics (Route B): cursor-less reduce (incremental key selection over the input delta); join via corgi `find` (find_ranges → equal-range merge-join, multi-record). as_collection reads columns directly. differential-dataflow proper: widen the JoinTactic/ReduceTactic/Fresh/ join_with_tactic/reduce_with_tactic seam from pub(crate) to pub, so an out-of-crate tactic can be implemented at all. (Candidate for upstreaming as the public extension point the TimelyDataflow#773 tactics were designed to be.) State: all 6 canonical programs match `vec`; 33 lib tests pass. reach ~1.9× slower than vec, compute-bound linear ~3× FASTER (columnar eval avoids the row backend's ~22% Value::cmp pointer-chasing). The remaining reach gap is entirely the reduce, still row-wise — the only operator not yet columnar. Notes / follow-ups: corgi_arrange.rs (old CorgiBatch impl) is superseded by corgi_chunk.rs and retained only as reference; scratch examples alongside the corgi_perf/corgi_progs/corgi_prof harnesses. Depends on frankmcsherry/wip corgi branch dd-arrange-api. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A standing harness that isolates each backend operator in a minimal DDIR
program, asserts corgi == vec, and reports the corgi/vec ratio against a
target. `PROG=<name> BACKEND=<corgi|vec>` loops one case for samply.
Baseline (n=100k linear / 20k arrange-y):
map8 0.30x ✓ (columnar compute, no row boundary)
filter 0.75x ✓
arrange 1.5x ✗ \
join 1.5x ✗ > every operator with a columns<->rows
reduce_distinct 2.0x ✗ / transcode boundary loses
reduce_count 2.0x ✗
reach 2.0x ✗
Corrects an earlier whole-program-profile conclusion ("only the reduce is
slow"): per-operator, arrange and join are independently ~1.5x — the shared
cost is the transcode boundary, not one operator. Beating vec means removing
row boundaries (chunk-native ingest, columnar operator emit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… vec
The columns↔rows transcodes at operator edges were self-inflicted, not
inherent. Removed all three:
- as_collection: read chunk columns straight into a CorgiContainer
(chunks_to_columns), no untranscode→from_updates round-trip.
- join emit: the projection already yields corgi columns; emit them via
give_container into a CapacityContainerBuilder<CorgiContainer> and drop the
JoinToCorgi unary — no untranscode + per-row give + re-transcode.
- arrange ingest: CorgiChunker (a ContainerBuilder) sort-consolidates each
input CorgiContainer's columns directly into CorgiChunks — replacing
ContainerChunker's drain-to-rows + VecMerger + row-based builder. It
ACCUMULATES to TARGET before consolidating, so it emits few large chunks,
not one tiny chunk per input container (else the columnar per-chunk set-up
dominates on many small batches).
Scorecard (corgi/vec): arrange 1.5x→0.9x (BEATS vec), join 1.5x→0.7x (BEATS
vec), reduce_distinct/count 2.0x→1.35x, reach 1.9x→1.45x. All 6 programs match
vec; 33 lib tests pass. Remaining reduce gap is the row-wise reducer (Rust over
Value rows) — next: columnar consolidate via corgi group→fold_add→filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
columnar_sum_by_key: the monoid Count/consolidate fold as pure corgi ops (group → map(fold_add) → filter via parse_ml + eval_graph), no Value rows. Signed diffs passed as raw two's-complement u64 bits; wrapping fold_add gives the correct i64 sum, `ne 0` the sign-agnostic zero-drop. Unit test: [(1,+1),(1,-1),(2,5),(3,9),(3,1)] -> [(2,5),(3,10)] (net-zero key dropped). Groundwork for the wave-based columnar reduce; not yet wired into the live reduce (which stays row-wise). Depends on corgi dd-arrange-api wrapping-Reduce. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…consolidation Count = Σ diffs ≤ t, and summing the consolidated per-value diffs equals summing the raw diffs — so for Count the consolidate_vals (a Value sort + clones) is pure waste. Sum diffs directly; no Value churn. reduce_count 1.39x→1.30x, all 6 programs match vec. Distinct/Min still consolidate per value (they reason about individual values) — that's the corgi two-level group→fold→filter path next; Collect stays row-wise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… tested) columnar_distinct_keys: the keys with a present value (net diff != 0, the nonzero simplification) as pure corgi ops — group by the (key,val) composite, fold_add raw diffs, drop net-zero, distinct keys of survivors. group-by-(key,val) is just group-by a projected composite; the nonzero test keeps everything on the raw-bit `ne 0` path (no signed encoding needed yet). Unit test: (1,10):+1,-1 → absent; (2,20),(2,21),(3,30) → present → keys [2,3]. Both monoid consolidate blocks now proven (sum for Count, two-level for Distinct). Remaining: the wave integration (feed many keys per wave, driven by the Rust time logic; reuse the existing delta/emit loop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…primitive Group a key column by identity, returning only integers: `perm` (row indices in group order) + `ends` (per-group exclusive end within perm). DD walks groups by index — group g is perm[ends[g-1]..ends[g]] — and drives its time/diff logic without ever seeing a key Value; keys and payloads stay columnar in corgi. This is the boundary model (corgi presents abstract int ids tied to times/diffs DD sees; DD hands back indices to include). Built on sort_perm + one batched compare_idx. Tested. Foundation for the path-2 columnar reduce (wave loop over group ranges). Note: cross-retire `pending` needs a STABLE key id (a u64 hash), since group indices are per-retire — converges with hashed-keys-as-u64. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oundary model Records: arrange & join now BEAT vec (transcode boundaries removed); multi-record primitives exposed; reduce consolidate blocks + group_offsets proven; and the design capture for a data-blind reduce tactic (framework presents as int-id/ time/diff, owns time navigation; backend supplies id-mapping + value callback; output ids are hashes — mint = hash the new value). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…flow#781) consumption plan Branch rebased onto int-proxy: the framework the SPIKE's boundary model called for now exists upstream (renamed ids -> int_proxy in review); the old in-branch copy is dropped. The hand-off section documents the current interfaces, the one-crossing-per-retire reduce_many contract, the assessment harness to replicate, the reduce-first plan, the traps already hit once (scan-presents, capacity leaks, giant-chunk settle), and the suitability questions the corgi agent should answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implement ProxyReduceBackend for corgi and swap CorgiReduceTactic -> ProxyReduceTactic. The DD tactic owns all time/lattice logic over (key_hash, value_id, time, diff); the corgi backend supplies only: - hashing: key_hash/value_id via corgi::arrange::hash_rows over the corgi key/val COLUMNS (columnar, content-addressed; DD never hashes). - reduce_many: ONE crossing per retire over every (key,time) bracket. Count/Distinct are integer-only (no value untranscode); Min/Collect gather+untranscode the bracket values once, never per key. Output ids minted by hashing the produced value column (agrees with present_output). - present_input/present_output: read arrangement chunks, restrict to changed keys by columnar semijoin (novel whole, history filtered); delta-proportional. - materialize: resolve ids -> real (key,val) rows, seal a CorgiChunk batch. corgi_progs: all 6 match vec (scc min/recursion + unnest collect included). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the DValue resolution maps (HashMap<u64,DValue>) with corgi COLUMN pools + integer id->row-index maps. The real keys/values never leave corgi columns: - present_input/output register representative key/val columns into per-retire pools (key_index/val_index -> offsets into concatenated blocks), no untranscode. - reduce_many builds each reducer's output value COLUMN directly: Count -> u64 prim, Distinct -> Unit, Min -> gathered input rows (id reused from input), Collect -> List. Ids minted by hash_rows over the built column. No transcode. - materialize resolves proxy ids to columns by gather + columns_to_batch (new corgi_chunk helper), column-native egress (no from_rows transcode). The only residual untranscode is Min/Collect's value ORDERING (the reduction contract is DDIR Ord, which is not corgi's structural order); the chosen rows are still taken columnar. Scorecard corgi/vec: reduce_distinct 1.01x -> 0.92-0.96x (BEATS vec), reduce_count 1.03x -> 0.90-0.93x (BEATS vec), reach 1.10-1.22x -> 1.06-1.13x. corgi_progs: all 6 match vec; interactive lib tests green (36). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the last untranscode (Min/Collect value ordering) with one columnar corgi sort_blocks per retire: - Min: gather positive-diff candidates, segment by bracket, sort_blocks -> each bracket's argmin = perm[block_start]; output row taken columnar, reusing the input value_id. - Collect: segment entries by bracket, sort_blocks -> sorted run per bracket, expanded by diff into a List column. Uses corgi STRUCTURAL order, which equals DDIR Ord for the non-negative scalar/tuple values these reductions see (all 6 programs); diverges only for negative ints (unsigned leaf compare) and list-valued compares (length-first), neither of which arises here — documented as an order invariant. Needs corgi::arrange::sort_blocks (exposed on fm/corgi dd-arrange-api 9b41cdc, the segmented sort re-exported from ops::cmp::order). The reduce backend is now entirely transcode-free. corgi_progs: all 6 match vec; lib tests green (36); reduce still beats vec (distinct 0.92-0.95x, count 0.93-0.94x). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
key_index/val_index keys are already well-distributed hash_rows u64s, so siphash on every register/lookup was wasted (~7% of the reduce tactic in profiling). Pass the id through unchanged. reduce_count 0.93x->0.88x; distinct/reach within noise (their id-maps are small). Gate green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The proxy records sort by (key_hash, value_id, time); key_hash is a full 64-bit content hash, so one MSD counting pass on its high byte splits into 256 near-uniform buckets, each finished by a comparison sort on the full key. One linear pass replaces most of the n log n (falls back to the plain sort for n<512 or clustered hashes). Profiling: from_unsorted 20% -> 8% of the reduce, driftsort gone. Wall-clock ~flat (0.89->0.90x) — the reduce is allocation-bound (malloc 33-38%), so this just uncovers that as the next lever. Framework (int_proxy + reduce_reference) tests green; corgi_progs all 6 match vec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
retire allocated fresh Vecs + an IdHistory for EVERY changed key (58% of the reduce's Vec-growth in profiling, IdHistory another 17%). Hoist the transient buffers out of the per-key loops and clear/reload them (IdHistory::load already clears+refills, retaining capacity): rep, raw_moments (drain), and phase-B meets, out_replay, emitted. Pure capacity reuse, no semantic change. Scorecard: reduce_distinct/count 0.90->0.85-0.86x, reach 1.06->1.00x (parity) at n=4000. Allocation was the real lever (sort CPU wasn't). Framework tests (int_proxy + reduce_reference) green; corgi_progs all 6 match vec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…IdMap) Phase B allocated a fresh BTreeMap per (key, moment) for the desired-vs-current delta. Replace with one HashMap keyed by an identity hasher (value_ids are already u64 content hashes), hoisted above the loop and cleared per moment (retains capacity, no per-moment alloc/free). Iteration order is irrelevant — deltas are consolidated and the output batch is sorted by from_unsorted. Scorecard: reduce_count 0.86->0.85x, reach 1.06->0.98x at n=4000 (below parity). Framework tests (int_proxy + reduce_reference) green; corgi_progs all 6 == vec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SCC is the sharp recursive case (nested fwd/bwd label propagation via min, negation, 4 joins). Findings: corgi is ~1.9-2.5x slower than vec and the gap GROWS with n (unlike reach, which reaches parity) — scc's nested-recursion / negation / min structure scales worse for corgi. And corgi DIVERGES from vec on larger random graphs (n>=1000: 1297 vs 1292) — a corgi-backend correctness bug (the proxy reduce tactic + reference backend pass the scc oracles in reduce_reference.rs, and it is NOT the min value-ordering — tested with a DValue-order argmin). Harness flags the mismatch instead of panicking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two debugging tools: - reduce_reference.rs: add reduce_int_proxy (int_proxy ProxyReduceTactic + Vec value backend over VecChunk, NO corgi) and scc_int_proxy_* tests that run SCC with only the reduce swapped to int_proxy, vs the cursor reduce. They PASS up to 50 nodes/120 edges/20 seeds — so the int_proxy reduce TACTIC is correct at SCC's nested-product-time depth. (reduce_reference's own scc_* compare cursor vs the *stock* reference tactic, never int_proxy — this fills that gap.) - interactive corgi_scc_min.rs: sweep+delta-debug an SCC corgi-vs-vec divergence to a minimal graph. Finds nodes=4, 5 edges [(0,0),(0,2),(1,3),(2,1),(2,2)]: corgi=3 vs vec=2. Conclusion: the divergence is in the CORGI backend, not the int_proxy work. The int_proxy tactic + Vec values reproduces SCC exactly; swapping in corgi is what breaks it. Next: narrow within corgi (reduce value backend vs join vs negate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ure) Four minimizer/trace harnesses (interactive examples): - corgi_scc_min: minimal SCC divergence = 4 nodes, [(0,0),(0,2),(1,3),(2,1),(2,2)]. - corgi_scc_trace: on that graph, export each SCC intermediate. Result: fwd::labels OK, trim_fwd OK, bwd::labels DIFF (node 2 -> label 1 in corgi vs 2 in vec), trim_bwd/scc DIFF. So the divergence enters at the backward min- reduce, structurally identical to the correct forward one. - corgi_cc_min: single-level fwd min-label propagation NEVER diverges (to 80 nodes) -> not single recursion. - corgi_bwd_min: two-pass fwd->trim->bwd with NO outer loop NEVER diverges -> not the chained nested scopes either. Conclusion: the bug REQUIRES SCC's outer recursion (the iterate wrapping fwd/bwd -> 3-deep product times, plus negation and feedback ). Symptom is a failed retraction (stale label survives). Suspects: corgi 3-level dynamic-time handling (leave_dynamic/results_in/enter_at) or negation under recursion. Not int_proxy (proven), not the reduce structure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
corgi_scc_stream: emit the full (data,time,diff) stream of SCC intermediates via inspect, per backend; diff consolidated by (data,time) to find the first moment of divergence (DD is deterministic at all times). Result on the minimal 4-node graph: fwd::labels and trim_fwd streams AGREE exactly (the 16-vs-10 raw counts were +1/-1 that consolidate away). The bwd min-reduce: its INPUT (proposals+nodes) diverges only at inner-times 258/513, but its OUTPUT diverges at 257/512 — one tick earlier. So at time (0,[1,257]) the reduce input agrees while the output diverges (vec retracts node2->1, corgi does not). By the inputs-agree-output-differs test, the bug is IN the corgi min-reduce. The int_proxy tactic is proven on SCC (scc_int_proxy), so it's the CorgiReduceBackend (present/materialize/changed-key), not the tactic. The forward min-reduce (same code) is correct -> a data-specific edge case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Instrumented CorgiReduceBackend::present_input (env CORGI_DBG) to dump node-2's presented input on the minimal repro, and compared to vec's true input stream: for (node2, label1), vec has [0,257]+1 [0,258]+1 then [1,257]-1 [1,258]-1 (added round 0, retracted round 1 -> gone); corgi's present has [0,257]+1 [0,258]+1 [1,258]+1 -- the round-1 retractions are MISSING (plus a spurious +1). And corgi's input STREAM (bwdin) agrees with vec at [1,257] (both -1), so the retraction enters the arrange but present_input never returns it. => corgi loses a reduce-input update when arranging/presenting under nested times; the min then reads a non-retracted label1 and stays stale = 1. Not the min logic, not the int_proxy tactic (proven) -- the reduce-input arrangement (CorgiChunk) or present_input's changed-key/novel-history read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For a primitive column — a bare Prim(64) or a 1-field Prod([Prim(64)]) — the value itself is already a collision-free id (i64 as u64 is a bijection), so pass it through directly instead of content-hashing. Compound shapes still hash. The id is used only as an identity for netting/dedup — its numeric order is never relied on — so the raw two's-complement u64 is correct even for negatives; no order-preserving swizzle needed. Applied CONSISTENTLY at every id site (both value presentations AND the reduce_brackets output values), so desired-current still nets by matching ids. Gate: corgi_progs 6/6 (all programs match vec). Scorecard: reduce_distinct 1.18x->1.07x, reduce_count 1.17x->1.07x, reach 1.26x->1.20x. join unchanged (the Phase 2a join is already hash-free). SCC flat within single-run noise (allocation-bound, not hashing-bound) — confirming columnar times is the SCC lever. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduces columnar times for CorgiChunk without a DD change. Our Time = Product<u64, PointStamp<u64>> already derives columnar::Columnar at every layer (Product, PointStamp, u64), with the container's Ref deriving Ord/PartialOrd (#[columnar(derive(Ord, PartialOrd))] on both Product and PointStamp). So the same per-tuple times that were Vec<T> (one PointStamp SmallVec per row — the dominant SCC allocation) live in <T as Columnar>::Container as SoA (offsets + values), one allocation pair per column instead of n. ColTimes exposes exactly what the CorgiChunk consumers need: push(&T) / push_ref (Ref straight across, no T), get(i)->T (materialize only for Lattice ops + emit), len, and cmp/cmp_cross (in-place ordering via the derived Ref: Ord, HRTB-bounded). Not yet wired — Inner.times stays Vec<T> this commit. Containment rationale: the Chunk trait boundary is whole-chunk + frontier-antichain only (CorgiChunk is Chunk but NOT NavigableChunk, so no cursor TimeContainer/TimeGat), and the batcher/spine move whole Rc<ChunkBatch> — so every O(data) per-tuple T is Inner.times, ours. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CorgiChunk's resident per-tuple times are now SoA (ColTimes over <T as Columnar>::Container) instead of Vec<T> — killing the per-row PointStamp SmallVec allocation that dominated the SCC profile (~40%). The hot maintenance paths (merge/emit/extract/advance/settle/concat) build and read times columnar: compares go through the derived Ref: Ord in place (no T materialized), range copies push Refs straight across, and an owned T is reconstructed only where a Lattice op is unavoidable (advance_by in compaction) or at the egress boundary (SortedRun for the join, CorgiContainer for as_collection — both want owned T). Bounds: a ColTime alias (Timestamp + Lattice + Columnar) bundles the constraint; the viral HRTB `for<'a> Ref<'a,T>: Ord` is discharged once in ColTime's blanket impl behind ColTime::cmp_refs, so downstream code needs only T: ColTime. Ingress paths (sort_consolidate, CorgiChunker accumulator, flatten_restricted) stay Vec<T> (they already hold owned T), converting at from_columns. No DD change. Gate: corgi_progs 6/6; 29 lib tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CorgiChunk::merge now drives off corgi `survey(kv1, kv2)`: one bidirectional gallop over the (key,val) columns yields Runs — maximal exclusive A/B ranges (bulk-copied, no per-row compare) + single Both matched pairs (time-ordered / consolidated here, since corgi owns no time). Replaces the per-pair two-pointer compare_at (the 22%-and-growing merge-compare tax). CorgiChunk::advance uses `group_bounds(ckv)` for the group-boundary scan (one pass) instead of per-row compare_at. Bumped corgi pin 9b41cdc -> f556868 (survey/group_bounds + hash_order reverted). Note on times: survey's Both is a positional (key,val) match, so it can transiently mis-order/under-consolidate the out-of-band times; advance re-sorts + re-consolidates each (key,val) group before any consumer reads the chunk, so it's correct end-to-end (gate 6/6 + 29 lib tests incl. merge/advance/is_graded oracle). SCC: 1.94x -> 1.62x vec (n=100k 6.55s -> 5.43s, ~17% off corgi). Branch corgi-survey-merge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tried a delta-proportional find_ranges seek of the changed keys (value-as-id makes them seekable in structural order). It REGRESSED SCC 1.62x->2.16x: SCC's changed set is broad (label propagation touches most keys each retire), so the scan already touches ~every row while the per-chunk gallop only adds overhead. Reverted to the scan; simplified the signature to `changed: &[u64]` and recorded the negative result in the doc so it isn't re-tried. Gate 6/6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ash_rows The `ids()` compound fallback now hashes via the canonical native `corgi::hash` (the designed boundary-id fold — width-blind, consistent-with-equality) instead of the branch-local `arrange::hash_rows` (width-seeded). Safe for DDIR: every leaf transcodes to u64 so width-blindness is a no-op, and there is no cross-path hash comparison (value-as-id for primitives and native hash for compounds are never used for the same value — shape is uniform per column). DDIR no longer references hash_rows, so the corgi side can retire it freely. Gate 6/6 (incl. compound-keyed adt/binders/unnest), 29 lib tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The survey merge (17ff06d) was a single-chunk MIRAGE. survey aligns only (key,val) (corgi owns no time), so its positional Both under-consolidates cross-side times; combined with consuming both chunks and no suffix push-back, it leaves canceling rows in the arrangement. Single-chunk (n<=100k) it looked like a win (SCC 1.94->1.62x, compare_idx 21.9->8.9%), but past the chunk boundary (edges e=2n cross TARGET=262144 at n~131k) the un-consolidated arrangement BLOATS every iteration and the reduce re-presents ever-growing input -> super-linear (n=150k: 104.9s / 19.5x vec vs 9.9s / 2.08x with two-pointer). Profile smoking gun: retire 96%, consolidate_updates 24.8% + quicksort 22.7% + collect_present 17% — the REDUCE drowning in bloated input, not the merge itself. Restored the two-pointer merge (full (key,val,time) consolidation + survivor push-back). Kept group_bounds in advance (perf-neutral but a clean one-pass scan; doesn't touch consolidation). Back to ~1.96x vec, LINEAR to n=250k. A survey merge needs a group-RANGE Both to compose with out-of-band times — flagged to corgi. Gate 6/6. corgi pin stays f556868 (group_bounds + native corgi::hash). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…c_big
Adds scc::strongly_connected_at(graph, logic) — the prioritized variant the
code comment always suggested — routing trim_edges through propagate_at so
label introduction is log-bucketed 256*(64-lz(logic(label))), byte-identical
to DDIR's enter_at($1[0]) delay in vec.rs/corgi.rs. strongly_connected is
unchanged (= strongly_connected_at(_, |_| 0)).
Oracle-tested: tests/scc.rs gains scc_at_{10_20_1000,100_200_10,100_2000_1}
running the prioritized variant against the sequential SCC oracle (incl. 1000
incremental rounds); all 6 pass.
corgi_scc_big gains the fair(enter_at) column. Measured (arm64, e=2n):
n=100k: native 4.16s, fair 0.62s (0.15x nat), vec 3.06s (4.95x fair), corgi 5.94s (9.61x fair, 1.94x vec)
n=150k: native 7.42s, fair 0.95s (0.13x nat), vec 4.83s (5.08x fair), corgi 9.90s (10.41x fair, 2.05x vec)
The compiled target was ~6.7x closer than the plain-native column suggested.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
E-graph-flavored identifier-lookup workload: parent forest (chains of CHAIN, roots self-looped) resolved to roots by an iterative min-join fixpoint (pointer doubling), then a term table (id,a,b) resolved by two integer joins against reps. Closed-form oracle (root = x - x%CHAIN) checked against vec, corgi asserted == vec, hand-written native twin oracle-checked too. Measured (arm64, chains=64, terms=2n): n=100k: native 0.59s, vec 2.34s (4.00x nat), corgi 3.49s (5.95x nat, 1.49x vec) n=1m: native 6.55s, vec 26.99s (4.12x nat), corgi 46.28s (7.07x nat, 1.71x vec) corgi-vs-vec gap GROWS with n (1.31x @1k -> 1.71x @1m) — profile candidate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same control structure as B1 but every node is a nested compound value (x -> (x>>10, x&1023), order-preserving): no value-as-id fast path — compound hashing, nested compares, and nested-column gathers carry the load. enter_at($1[0][0]) buckets by group; the fair native twin is strongly_connected_at over (i64,i64) with logic |n| n.0 (identical delay). Correctness: corgi == vec on the full SCC edge set; compound output mapped back through the encoding == the plain-encoding program on the same graph. Measured (arm64, e=2n): compound keys hurt everyone, corgi worst: n=100k: fair 2.76s, vec 21.56s (7.80x), corgi 61.42s (22.22x fair, 2.85x vec) n=150k: fair 5.22s, vec 41.48s (7.95x), corgi 132.77s (25.43x fair, 3.20x vec) (B1 plain-int corgi is 1.94-2.05x vec — the compound-key path is corgi's weakest spot so far, and the ratio grows with n.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per row (a,b < 2^15): 8-element arithmetic list, flatmap explosion, Fwd/Bwd variant + case + 4-element fold per element, min per 16 buckets (min so the compute can't be dead-coded; reduce stays tiny). Closed-form Rust oracle checked against vec, corgi == vec, native twin oracle-checked. Values constrained non-negative + non-overflowing: the first draft (values to 1e6, a-b negative, e*e overflowing) exposed corgi min DIVERGING from vec on negative ints — corgi's structural order is unsigned at the integer leaf. Real abstraction gap, logged in the read-out; out of contract for B4. Measured (arm64): corgi is NOT columnar on this path — flatmap/case/fold run the row-wise fallback (ir::eval + transcode), so the expected compute win is absent: n=1m: native 1.09s, vec 8.35s (7.69x nat), corgi 8.89s (8.19x nat, 1.07x vec) n=4m: native 4.55s, vec 34.61s (7.60x nat), corgi 37.19s (8.17x nat, 1.07x vec) Columnarizing flatmap/case/fold is a top lever candidate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Loops one backend of scc-compound at a fixed size for samply, built identically for apples-to-apples profiles. Iteration-1 profile findings (n=50k, corgi 28.0s vs vec 11.4s): - retire subtree = 77% of corgi; vec's Value::cmp/partial_cmp/eq = 26% self. - consolidate_updates_slice_slow 17.6% total. Two tactic-side fixes (permutation sort; interned-time integer consolidation + stable run-merge) were both measured FLAT and reverted: the hot consolidation is NOT the tactic's delta emit — it is next_window's presentation build in corgi_reduce_backend (collect_present clones an owned PointStamp per presented row, then consolidate_updates comparison-sorts the AoS records; the caller frame was inlined away). That backend site is the next lever. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
B4 corgi profile (n=1m): ir::eval 19% (row-wise List/Case fallback), Value churn ~15%, arrangement merge of the 8m exploded rows ~29%, reduce 14%. Both backends run B4 row-wise — the columnar win is unrealized here. Iteration-4 lever scoped in the read-out: columnarize List/FlatMap/Case in the backend via existing corgi kernels (Iota/Gather/Flatten/Fold/Inject/ MapSum + gather_lanes interleave). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
B3 corgi profile at n=1m surfaces two concentrated corgi-kernel buckets the SCC profiles never showed: sort_leaf_blocks 17% self (discrimination-sort leaf pass, under ingress sort_consolidate + materialize from_columns) and the join's find_ranges probe 17.8% total (interpreted CmpOp::eval compare per binary-search step). Both kernel-general levers; details in the read-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 timing runs at n=100k (fixed graph): corgi 3.76-3.89s, vec 2.97-3.03s, ratio 1.26-1.30x; n=250k: ratio 1.28-1.29x. 3 graph instances (SEED=1, 12345, 999983): ratio 1.24-1.29x, vec/fair 4.78-4.94x. Both axes ~ +/-2%; the reported ratios are solid to about two figures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New native_scc_prof (fair enter_at compiled twin). Triptych at n=100k, self-time by module: allocator native 2.2% / vec 28.4% / corgi 27.2%; memmove native 0.7% / vec 2.8% / corgi 11.9%; timely progress tracking ~0.5% in BOTH backends. The 4.8x shared tax is BUFFER LIFECYCLE (allocation + copy churn), not progress tracking (kills lever 1b as a priority), not operator count (rung B), not edge time repr (rung A). vec's churn is per-row ir::Value boxes (cmp+clone+eq ~20% self); corgi's is column rebuilds (fresh Arc<Vec> per gather/merge/present). The suite ran on the SYSTEM allocator (only ddir_server/ddir_vec linked mimalloc). Adding mimalloc to all suite + prof binaries (one binary per benchmark: every column shares it; native columns move <5%, DDIR 12-36%): B1 250k: corgi 8.67 -> 7.07s (0.96x vec, 4.29x fair; was 5.2x fair) B2 150k: corgi 53.45 -> 42.94s (1.22x vec) 100k: 1.08x vec B3 1m: corgi 27.24 -> 19.79s (0.82x vec, 3.17x fair) B4 4m: corgi 20.09 -> 13.00s (0.45x vec, 2.96x fair) Gate 6/6 incl. fusion oracle; [checked] rows green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pin moves from f556868 to the dd-arrange-api tip, which carries the four landed kernel additions (wip#7). Gate green; workspace passes -D warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # differential-dataflow/src/operators/int_proxy/history.rs # differential-dataflow/src/operators/int_proxy/join.rs # differential-dataflow/src/operators/int_proxy/reduce.rs # differential-dataflow/src/operators/mod.rs
Ten purpose-built binaries (four benchmarks + six profiling twins) collapse into one runner: interpreted programs live in examples/programs/*.ddp where they belong; the irreducibly compiled parts — input generators and the native twins, including the enter_at-fair SCC — register in the runner by bench name. BACKEND=/ITERS= select profiling mode (loop one column, a samply target), replacing the prof twins. Output formats, env protocols (N/SEED/CHECK/CHAIN), and all correctness assertions are unchanged; all four benches verified [checked] plus a prof-mode smoke; gate green; workspace passes -D warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eleven milestone/experiment binaries (rungs 0-4, M1-M4, Q3 of the backend build-up) whose questions are answered and whose coverage the survivors carry: the gate (corgi_progs) runs reach/scc and richer programs continuously, the scorecard owns per-operator triage, and ddir_bench owns end-to-end measurement and profiling. SPIKE.md narrates the arc they documented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The .ddp extraction overwrote the pre-existing gate program with the benchmark's embedded SCC (different export shape, comments lost). The gate kept passing because the benchmark program is also a valid corgi==vec check — which is how it slipped through. scc.ddp is restored byte-identical; ddir_bench loads scc_bench.ddp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The five crate-root corgi_*.rs files become src/corgi/{container,chunk,join,
reduce,logic}.rs. No code change beyond the module paths: container is the
CorgiContainer on dataflow edges (was corgi_backend.rs), chunk the columnar
Chunk + arrangement plumbing, logic the DDIR-term compiler, join/reduce the
tactic bindings. backend/corgi.rs (the Backend impl, parallel to backend/vec.rs)
and col_times.rs (a generic columnar-time utility that imports no corgi) stay
where they are. Gate 6/6 both tactics; workspace passes -D warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses review of the subsystem PR: - corgi_progs (the corgi==vec gate) moves from examples/ to a tests/ integration test (tests/corgi_backend.rs) — it is a gate, so its home is cargo test / CI, not examples/. Runs the six canonical .ddp programs, one #[test] each. - Remove the benchmark/measurement tooling that isn't a DDIR example: ddir_bench, corgi_scorecard, and the five benchmark-only programs (ast/ident/scc_bench/ scc_compound/scc_plain .ddp — the latter three the duplicate-SCC sprawl). These are dev tooling, not part of the library; they stay on the dev branch and can land separately if the later perf PRs want reproducible in-tree numbers. - Drop the working notes (SPIKE.md + two findings files, ~970 lines of markdown). examples/ is back to DDIR programs plus the pre-existing server/dump binaries. strongly_connected_at stays (a general algorithms/ addition, tested by tests/scc.rs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
col_times exists to serve CorgiChunk (its own doc says so) and is used only by the corgi backend's modules — it belongs in src/corgi/, not at the crate root. No code change beyond the module path (crate::col_times -> crate::corgi::col_times). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the local-dev path note and the reference to a findings file that no longer exists; keep the git rev pin (explicit and reproducible) with a one-line purpose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The header described an early milestone: Spine<Rc<CorgiBatch>> (the arrangement is a ChunkSpine<CorgiChunk> now), reduce 'awaits' a retire design (it is implemented via the tactics), and a 'this iteration ... = todo!()' note (nothing is todo). Rewrite it to describe the current, complete substrate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CC/Row/Upd/CTrace were defined ~110 lines below apply_ops, whose signature reads in terms of CC. Move them to just after the imports so the shorthands are in scope where the reader first meets them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Explain the empty enum: it is never a value, only a type carrying the Backend impl (selected via render_tree::<CorgiBackend>); the empty enum makes it unconstructable. Mirrors VecBackend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.