Migrate SCM+Sync into package code, harden it, stand up an internal relay - #5692
Draft
StachuDotNet wants to merge 1 commit into
Draft
Migrate SCM+Sync into package code, harden it, stand up an internal relay#5692StachuDotNet wants to merge 1 commit into
StachuDotNet wants to merge 1 commit into
Conversation
StachuDotNet
changed the title
StachuDotNet
changed the title
StachuDotNet
force-pushed
the
kernel-substrate
branch
3 times, most recently
from
August 9, 2026 01:31
002c821 to
f7b70be
Compare
…elay
The op log itself is unchanged: `package_ops`, content-addressed ids, origin_ts
LWW and the fold are where they were. What moves is who owns them, what a branch
is made of, and which decisions survive leaving one machine.
Sync, branches, conflicts, review and propagation were F#. They are Dark now.
The F# that remains does what only F# can do (parse, hash, serialize, execute,
store bytes); everything that makes a DECISION moved up into package code.
Most of the SCM layer in `LibDB` was a second implementation of decisions Dark
could make for itself, so it got deleted rather than ported: keeping `Merge.fs`
and `Rebase.fs` would have meant two places deciding what a merge means.
What to check:
1. `package_ops` is the only source of truth. Everything else -- `locations`,
`package_functions`, `propagation_policy`, conflicts -- is a projection you
can drop and re-fold.
2. A branch is an overlay, not a copy. Its ops live in the same table,
`effective = 0`, tagged in `op_branches`. `locations` has NO `branch_id`, so
a read that goes straight there answers about MAIN while you stand on a
branch, plausibly, which is why it is hard to spot.
3. Convergence is deterministic: origin_ts LWW with a hash tie-break, computed
identically everywhere.
4. Decisions travel. A conflict resolution, a propagation pin, a constraint ack
and a merge are all ops, so they sync and hold rather than being local
opinions.
Squashed from 96 commits and rebased onto main. The individual messages are
preserved on the `pre-rebase-backup` ref if the reasoning behind a particular
change is wanted.
Full notes, including a before/after table and what is left, in
`notes/fresh-arch/PR-kernel-substrate.md`.
---
Rebasing over the perf work needed six real decisions rather than mechanical
conflict resolution, recorded here because each one is a place the two branches
disagreed about the model:
- **`isHarmful`**: upstream made it synchronous (dropping a Ply bind per package
call); this branch dropped the `BranchId` (a deprecation is keyed on content,
and content means the same thing on every branch). Both are wins and they are
independent, so it is now `FQFnName.Package -> bool`.
- **`withCache`**: took upstream's shape (caching the `Some` wrapper so a hit
hands back the same object) plus this branch's registration in the
invalidation list, which is the whole reason that list exists.
- **The dependents query**: upstream added `ORDER BY l.owner...` and a
`branch_id` filter. This branch had deliberately removed the `locations` join
there -- a branch's items have no rows in `locations`, so joining makes every
branch-authored dependent invisible to propagation. Kept it unjoined and
ordered by hash, and said why in the SQL.
- **Store reconciliation**: upstream reconciles by release number
(`planCliUpgrade`). That footing is gone here, so this keeps the top-up path
and takes upstream's `timed` wrapper around it.
- **Branch ids are Strings**, not Uuids: twelve declarations across four
package files came over from upstream typed `Uuid`.
- **`InterpreterStatsSink`** was defined on both sides; kept upstream's, which
bounds the bag.
One upstream bug fell out of it, caught by this branch's command-surface sweep:
`dark view <nonexistent>` interpolated an `Option<String>` directly, so the
error path meant to name your target crashed instead of printing it.
Their perf work is worth having: the test suite went from 5m16 to 3m24 on this
box, though about 50s of that is CliTraces leaving the default suite in the same
rebase.
---
Four bugs found by running this against the relay deployed on Fly rather than
against a test double. All four were SILENT, which is the failure mode this
design makes easy: content addressing means a wrong answer is usually a
plausible one.
- **An upgrade skipped the ops a client had already pushed.** Op ids are content
hashes, so a client pushing its tree lands the ids the binary's own seed
carries. Hosted ops are stored inert; the seed top-up used `INSERT OR IGNORE`,
which skipped them as already-present and left them that way, so the fold
never ran over them. Names resolved, content behind them did not, and the
store died at startup on a hash in its own log. Ops the seed carries are now
woken rather than ignored.
- **Push could go quiet.** The push cursor is a local rowid, which means nothing
against a store that has been re-grown; the cursor then points past the end,
every push finds nothing after it, and it says "nothing new to push" while the
relay stops receiving. On the store this was found with, that was 5,119 ops
never sent. A cursor past the end of our own log is no longer treated as a
position in it.
- **`sync status` contradicted `sync`.** It compared the two stores' newest
timestamps, but the same code written on two machines lands the same op id
carrying each machine's own origin_ts, so two identical stores report
different maxima. Status now answers from the same two cursors sync skips on.
- **Every function page on the relay was a 500**, from a uuid handed to a
parameter wanting a string. Nothing had ever called those handlers; the relay
had tests on its wire codec and none on the half that renders. It has both now.
The performance claim for the push cursor was also wrong and is corrected: an
idle sync on a 12k-op store is ~6.4s before and ~1.1s after, not the 79s quoted
in two comments. That number came from a container with tracing on, a trap
already written up in `notes/fresh-arch/SYNC-PUSH-PERF.md`.
A store that can't be written to now says so. A read-only mount, a store owned by
another user and a full disk are conditions of the machine, not defects, and each
one reached the user as a .NET stack trace naming neither the store nor the
problem -- writes went through `Result.unwrap`, which prints the raw exception to
stdout and raises "TODO: failed to unwrap". Those three SQLite codes are now
translated at the one place every query funnels through, and carried as a
`StoreConditionException` rather than an `InternalException`, since the right
response is the opposite one: nothing to report, nothing to fix in the code, and
the person running the command can resolve it once told which it is.
An unopenable store went from an exception tree plus stack to a single line. A
read-only store went from 35 lines on stdout to none, and the sentence that says
what to do is now the first thing on stderr rather than buried.
`scripts/run-in-docker` also grew a guard on stdin. It reads fd 0 to completion,
which assumes fd 0 will reach EOF; hand it something that never does and the
pipeline deadlocks after the `docker exec` has already exited, leaking processes
on every call. Pipes and regular files are forwarded as before.
`dark branch pull --from <who> <url>` took the url only on the other side of the
flag, and the order it did not take fell through to the help text -- which reads
as though nothing happened, on a command that was right apart from word order.
Both orders work now, and an unrecognised form says which form it did not
understand before printing help, rather than printing help alone.
Archiving a branch now travels, which it was documented to do and did not. The
op type had an `Archived` case, the serializer wrote it and the fold applied it
idempotently -- but nothing anywhere authored one, so the only event ever written
was `Merged`. A branch you archived stayed on your colleague's machine looking
like live work forever. Caught by archiving three test branches and watching the
other machine keep listing them.
A Release CLI shipped the developer's working store. `dotnet publish -c Release`
gzipped `rundir/data.db` straight into the binary, and that file holds sync state
in the same place as packages: a fresh install came up carrying the relay WRITE
SECRET in plaintext, the building machine's sync identity, its branch names, and
cursors saying everything had already been pushed and pulled. So a new install
announced itself as the machine that built it and went silent against a relay
holding 42,000 ops it did not have.
`export-seed` already knew all of this -- `LibDB.Seed.export` strips config,
sync bases, branches, op ownership, conflicts and traces, and explains each one.
Only the self-contained release script ran it. So the fix is wiring, not a second
implementation: the project embeds `rundir/seed.db`, and the Release build fails
with the command to produce one if it is missing. Building a seed no longer
overwrites the working store either, which the release script used to do as a
side effect.
Found by making a genuinely fresh instance instead of reusing a rundir, which is
the only way any of this is visible.
The pre-upgrade backup is now taken only when the top-up will actually change
something. It ran on every first launch, copying ~80MB to protect a store from an
upgrade that had nothing to do, and announcing an upgrade that was not happening.
The repair paths still back up first: verified for a store missing ops and for one
holding ops that were present but inert.
A wrong write secret told you to supply a write secret. The relay answered the
same 401 whether the header was absent or simply wrong, so `dark connect --secret
<typo>` came back advising `--secret`, which sends you looking for the flag you
just used. Only the relay can tell those apart, so it now says which one happened.
Tests cover the fail-closed case in both directions: an unconfigured relay refuses
`/sync/push` AND `/branch/push`, since guarding one and forgetting the other is
the easy mistake.
The relay is a package browser now, not a landing page with two pages behind it.
Modules list their submodules and group their contents by kind; every item shows
its doc comment, its source, what it uses and -- the half you cannot get from
reading the code in front of you -- what uses it, from the fold's own call graph.
Search, a publisher index and a stats page, breadcrumbs on every deep page, and a
front page that leads into the packages rather than describing them.
Doc comments render their markup (`{{code}}`, `<param x>`, and the three other
tags that actually occur) instead of showing braces and escaped angle brackets.
Escaping runs first and the markers are matched in their escaped form, so a doc
comment cannot introduce markup of its own.
The search box is the only place on a read-only site where a visitor chooses what
goes on the page, so there is a test that it stays escaped.
`sync status` no longer nags after a sync in either direction. A pull raised the
op count without moving the pushed stamp, so it reported "ops to send" and a push
that sent nothing; a push moved the relay's head without recording that we had
seen it, so it then reported "ops to receive" and sent you to pull back the ops
you had just sent. Both halves record what they now know.
The relay's pages say which of its two numbers is which. It holds ~70,000 ops and
serves source for ~6,000 items, because a relay stores what you push WITHOUT
folding it into its own package set -- which is what makes it safe to point a
machine at, since pushing code to a relay cannot change what that relay runs. The
consequence is that hosted code is held and handed back but not browsable, and the
pages now say so instead of implying the ops produced the items.
Comments cut back to the point. This branch had been adding roughly three comment
lines for every net line of code, much of it narrating how a thing came to be
written rather than saying what is true: what the code used to do, what was found
by doing what, which test could not see it. A comment is read by someone deciding
what to do next, so that all goes and the invariant stays.
Six doc comments had come unstuck from what they document, describing functions or
tests that were defined further down or, in one case, not at all -- 98 lines of it
stacked in one place in the CLI trace tests. Each is back on the thing it describes,
and the orphan is gone.
Section headers are plain ASCII, and `exportMainOps` is gone -- superseded by
`exportMainOpsSince` when push grew a cursor, and called by nothing since.
`dark deprecate` under-reported its own collateral on a branch. Deprecation is keyed
on CONTENT, so it hits every name holding that hash, and the command warns you which
ones before asking. That warning read `locations` directly. `locations` used to carry
a `branch_id` and the query filtered on it; dropping the column for the overlay model
left the filter with nothing to bind, so on a branch it listed main's names and
silently omitted the branch's own. Now in `SCM.PackageOps.namesAtContent`, which
unions main with the branch frontier, and the CLI writes no SQL for it.
`deploy/fly.toml` no longer names the app. A Fly app's default hostname is
<app>.fly.dev, and the relay's read endpoints are unauthenticated, so naming it in a
public repo publishes the address and everything on it. Deploys pass `-a`.
The relay's one remaining string-concatenated query is a bound parameter. Its value
came from the store rather than a request, so it was not reachable, but a concatenated
predicate does not belong in the one file that faces the internet.
A test now reads the source and fails if any Dark file outside the SCM silos queries
`locations`. That read is main-only and answers plausibly on a branch, so the failure
is invisible at run time on a single-branch store; a static check is the only kind that
catches it. `matter.dark` is exempt -- a relay holds no branches, so main's projection
is its answer.
The relay highlights source. Same semantic tokens the CLI colours, so a name reads
the same in a terminal and on a page, with the token types mapped to CSS classes
instead of ANSI. Escaping runs per SEGMENT rather than over the whole line, because
escaping first moves every column after the first `<` and each token would then wrap
the wrong characters. Source that will not parse still renders, escaped. Doc comments
are marked here rather than by the tokenizer, which does not emit a token for them.
An edit of yours now survives an upgrade that ships a different version of the same
name. The bindings the build did not write are recorded before the top-up folds and
re-asserted after it, for the names the fold actually moved. The re-assertion is a
`Resolve`, since re-authoring the original `SetName` dedups and folds nothing, and
since editing `locations` directly would be undone by the next re-fold.
Which bindings those are needs PROVENANCE, and `locations.op_id` against the current
seed does not give it: a store grown from an earlier build is full of ops that build
shipped, and the next build ships different ones for everything that changed. Asking
only "is this op in the seed I hold" therefore calls the entire previous package set
locally authored, and "keep your edits" becomes "pin the store to the old build" --
measured at 649 names on a store with no edits at all. So `seed_ops` records what each
build's seed carried, filled with certainty when a store is first created and appended
on every top-up. A store predating the ledger declines to keep anything once, rather
than guessing.
Verified across two builds whose seeds differ: a store with no edits takes the new
version silently, and a store with one takes it back and says so. It names what you
edited and counts what followed separately, because one edit to a core function
repoints hundreds of callers and a list led by those reads like a disaster.
`seed_ops` is declared where it is first used rather than only in `schema.sql`.
Resource extraction runs before migrations, so a store made by an earlier build
reaches the ledger query without the table existing, and every startup is a crash
on a store that was working a moment earlier. Declaring it at both points that
touch it covers a fresh store and one that predates it.
Branch reads on a relay now take the same secret as writes. `/branch/list` and
`/branch/pull` were open, so anyone who knew the address could enumerate a
colleague's branch ids and pull the bundle: name, ops and content blobs of work
that has never been published anywhere. That is the one thing on a relay that is
categorically not public, and it was the one thing with no gate.
The package log stays open. `/m` and `/p` already serve it as HTML, so gating
`/sync/pull` would protect nothing.
Reads carry the secret in a header, which needed `httpGetUnsafeBytesWithHeaders`:
the existing unsafe GET takes no headers, and a query string ends up in access logs
and proxy traces. Tests cover both endpoints refusing, and that a refusal does not
echo back what it was asked about.
`package-ref-hashes.txt` goes back to being untracked, as it is on main. This branch
had re-tracked it, on the argument that `kernelHash` should fingerprint the build
rather than whatever a machine's store held. That argument is real but it loses to
the practical one: two branches that both touch packages produce conflicting hashes
in a generated file, and a rebase on GitHub cannot regenerate it, so every such
branch needs a manual pull-reload-commit. The MSBuild target that creates an empty
file on a fresh clone is untouched and still covers that path.
The build payload for the relay image moved to `rundir/deploy`, which is already
ignored, so the repo's ignore file no longer carries a line for a build artifact.
The image's build context is now that directory.
StachuDotNet
force-pushed
the
kernel-substrate
branch
from
August 9, 2026 02:35
f7b70be to
14803bc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ownership
Sync, branches, conflicts, review and propagation were F#. They're Dark now. I deleted the F# rather than
porting it --
LibDB/{Merge,Rebase,Conflicts,Lww,Resolutions,ChangeSummary,BranchOpPlayback}.fsand thematching builtins -- because keeping both would leave two places deciding what a merge means.
packages/darklang/scm/is a silo per table, each owning every statement against its own. F# keeps thefold, authoring, the merge mechanism, the cascade, and schema changes to canonical tables. The rule I held
to: decisions in Dark, mechanism in F#.
canMergeis Dark; flipping the frontier is F#.A branch is an overlay now
BranchOpand its playback are gone, as arepackage_ops.branch_idand thebranch_opstable. A branch'sops sit in
package_opsateffective = 0, tagged inop_branches. It used to be a column value; it's aset of ops layered over main.
effectivesplitting stored from applied is what lets a branch, a review queue and a relay be onemechanism instead of three.
The cost, and I'd flag it in review:
locationslost itsbranch_id, so a direct read answers about mainwhile you're on a branch, and answers plausibly. The schema used to protect you there and now discipline
does. There's a test that fails if any Dark file outside the silos reads it.
Decisions travel
New op cases:
Deprecate,Undeprecate,Resolve,Decide,BranchEvent.PropagateUpdateandRevertPropagationare gone.That's the substance behind "decisions travel". A resolution was a row in
resolutions, a conflict a rowin
sync_conflicts, and rows stay on one machine. As ops, a conflict resolution, a propagation pin, aconstraint ack and a merge all sync and hold.
Resolvehas to be its own case because ops are content-addressed: re-authoringSetName(name -> old hash)produces an op that already exists, dedups, and folds nothing. There's no way to say "the same binding, but
now I mean it" as a plain op.
SetNamegainedprevious, the hash it replaced. Without it, "we both invented this name" and "I editedyour version" look identical, and only the first is a conflict.
Commands
Newly at top level:
pushpullconnectwhoamiidentityswitchbranchesdifflogresolvepropagateconstraintsackundeprecate. Nothing removed.Less new than that list looks. Propagation already ran on every edit --
cli/packages/propagate.darkwasinternal helpers; what's new is a policy you can set.
Constraintswas already named inProgramTypes.fs;this implements it.
syncexisted and now means push-then-pull against a relay.Actually new: the relay, per-instance identity, and
resolve/ackas durable syncable answers instead oflocal state.
Branch names are aliases over uuids. You and I can each hold a
fix-auth; pulling mine imports it asfix-auth~<shortid>.Propagation
Cascade unchanged, policy is new.
propagate pin <name>/follow <name>write aDecideop. Both areretroactive -- by the time you pin, the repoint already happened, so pin undoes it and follow catches you
up. Most-specific-first (item, module, parent modules, owner); no row means follow.
Un-propagated usages are the steady state in a distributed store, not a backlog: a caller on your machine
can't be propagated to from mine, and mustn't silently repoint when we sync. So staleness is
dark constraints, not a line at commit that reappears forever.A pin travels. You pinning a caller rolls it back on my machine too. That's the design working as
written, and it's the piece I'd want us to agree on before we lean on it.
What it's like to use
Conflicts don't block. They're auto-resolved by LWW and recorded, so the loser is kept and reviewable, and
the record names the person rather than "yours"/"theirs".
Failure paths say what to do: a network failure tells you the work is local and safe, a read-only store or
full disk says which and where, a wrong relay secret reads differently from a missing one, and a command
given a target it can't find names the target. Tab completion covers the SCM commands, and anything that
answers a question answers
--json.The relay
packages/darklang/matter.darkis a relay and a package browser, served bydark serve Darklang.Matter.router, with the image andfly.tomlunderdeploy/. It's deployed andrunning for internal use. The address isn't in the repo: its read endpoints are unauthenticated, so treat
anything pushed there as public. Ask me for the host and the write secret.
It stores and hands back ops and never folds them into its own package set, which is the load-bearing part:
pushing code to a relay can't change what the relay runs.
The browser reads the relay's own store -- module tree, source, doc comments, and what-calls-this from the
fold's call graph, plus search.
/p?name=Darklang.Stdlib.List.mapis the one that shows it off.Only the relay's own package set is browsable, since only it is folded. Showing the versions clients have
pushed needs a second projection folded for display.
Separately, and not in this PR: darklang.com's package pages were pointed at endpoints that no longer
exist. Fixed in that repo as one local unpushed commit, so nobody redoes it.
Ready to use?
For the two of us plus the relay, yes for what's below, with one gap I'd close first.
Upgrade is newest-stamp-wins, not "local edits win". Edit a stdlib function today, upgrade to a build cut
tomorrow, and the build wins silently.
locations.op_idrecords which op wrote each binding, so the fix isavailable; it didn't block the relay, which never authors.
Also open: a branch rename doesn't travel, though merge and archive do. Hosted ops aren't browsable. Op ids
are a 128-bit truncation of the content hash.
Testing
The suite covers the fold, the overlay, conflict detection and the CLI surface. Past that I ran it for real
against the deployed relay with fresh instances, which is how the worst bug turned up:
producing a real one that's auto-resolved, kept, and overridable back across
secret, no secret, dead relay, older client against the upgraded relay
The bugs I found were all silent -- no error, just a plausible wrong answer. That's the failure mode
content addressing makes easy and the thing worth looking for. The worst: a Release CLI embedded my working
store, so a fresh install carried the relay write secret in plaintext, claimed to be my machine, and went
quiet against a relay holding everything it was missing.
export-seedalready stripped all of that; onlythe self-contained release script ran it. Fixed as wiring, the build now refuses without a seed, and I've
rotated the secret.
Where to look
LibDB/PackageOpPlayback.fsfor the fold,LibDB/Branches.fsfor the overlay and the trap above,cli/sync.darkwithscm/packageOps.dark's transport section for the wire,scm/branches.darkfor what amerge is allowed to do.
One thing I chose not to do:
scm/packageOps.darkspans identity, decode, preview, reconciliation, draft,wire and relay storage, and wants splitting. It's a rename across a lot of call sites and I didn't want to
land that unreviewed.