fix: fsync before acking durable writes so the server can't report content it never persisted

[lehuan5062]

Problem

A commit published a revision referencing content that existed nowhere: not on the remote, and not in the committing machine's local pack storage either, despite that machine's store index flagging the fragment PayloadStoredDurable. This is real data loss, not a bookkeeping error.

Root cause is on the server. handle_put does await ImmutableStore::put before acking, but the local packstore write path underneath it (PackStore::store) writes payload bytes with a plain positional write and never calls fsync. The only automatic flush, flush_delayed on a 5s timer, explicitly skips fsync as well; a real sync_data = true flush happens only on graceful shutdown or GC. So a put is acked as durable while the bytes exist only in the page cache, the client discards its local copy on the strength of that ack, and a crash anywhere in that window loses content that was reported durable.

A second, independent failure mode sits on the client: leader_body swallowed upload errors, so an upload that failed or timed out still let the commit proceed to branch publish.

Change

Sync before the ack, batched rather than per-write:

• lore-storage: a new per-store background ticker (durable_flush_loop) fsyncs every group's packstore on a fixed interval and bumps that group's FlushBarrier generation. A store() call writing fresh payload bytes under a PayloadStoredDurable claim snapshots the generation and waits for the ticker to pass it, so there is no sleep and no spawned task in the write path itself.
• FlushBarrier::wait_past waits for the generation to advance by 2, not 1. A tick already in flight when the writer sets its dirty bit can complete without having observed it; ticks are sequential, so at most one can be non-covering and the next is guaranteed to sync the write. Waiting for +1 would reintroduce the bug on that interleaving.
• PackStore::flush_all is cheap when clean (one compare_exchange per packfile, no I/O), so the ticker walks all groups every tick without per-group pending-writer tracking.
• New durable_flush_tick_ms setting. 0 disables the ticker entirely and is the default, so clients that only cache a remote-durable copy pay nothing; lore-server sets 15ms, bounding the worst-case unsynced window to roughly 30ms.

Client write path, for the swallowed-error mode:

• leader_body now returns the upload error instead of discarding it, after still writing the local entry non-durable with the payload cached as a safety net, so a failed upload fails the commit before branch publish.
• remote_put_retry bounded-retries Disconnected in addition to SlowDown, invalidating the session between attempts.
• follower_future takes a require_durable flag so a follower whose own write is remote-coupled does not accept a local-only entry as success.

Recovery for content that is still cached locally but was never confirmed durable:

• New lore repository push-content command (explicit addresses, or --scan to sweep the local store), and repository verify extended to check content durability against the remote rather than structural integrity alone.

Validation

cargo test -p lore-storage -> 169 passed, including two new regression tests: durable_put_waits_for_batched_fsync_before_returning (a durable write does not return until the ticker has confirmed it) and non_durable_batch_window_skips_fsync_wait (tick interval 0 spawns no task and adds no wait).
cargo check --workspace --all-targets -> clean.
cargo clippy --all-targets -- -D warnings --no-deps -> clean; cargo +nightly fmt --all --check -> clean.
Smoke tests: scripts/test/test_push_content.py -> 2 passed; scripts/test/test_verify.py -> 11 passed as a regression check.
docs/reference/lore-cli-commands.md regenerated from lore --markdown-help. That diff also picks up unrelated drift left by an earlier commit (--gc renamed to --no-gc, new --cache flag).

Not covered, stated plainly: the original incident's address is unrecoverable and no test restores it. There is no crash-injection test that kills the server inside the unsynced window, so the regression tests assert the barrier ordering that closes the window rather than the crash itself.

AI assistance disclosure

I used an LLM (Claude Code) to help with implementation, code comments, and documentation on this change, and to help word things clearly as a non-native English speaker. I reviewed and tested all the code myself for its correctness, security, and license compliance.