Skip to content

feat(mem_wal): observe flush latency through the metrics facade - #8625

Open
hamersaw wants to merge 4 commits into
lance-format:mainfrom
hamersaw:feature/mem-wal-flush-metrics
Open

feat(mem_wal): observe flush latency through the metrics facade#8625
hamersaw wants to merge 4 commits into
lance-format:mainfrom
hamersaw:feature/mem-wal-flush-metrics

Conversation

@hamersaw

Copy link
Copy Markdown
Contributor

What

WriteStats tracks flush counts and cumulative time, but a running total cannot be resampled into a distribution — the individual observations are gone by the time anything polls it. An embedder can compute an average and nothing else, which is the wrong shape for latency: a flush pipeline is judged on its tail, not its mean.

This observes each flush individually through the metrics facade that lance-io already uses for object store operations. Observations route to whatever Recorder the embedding process installed, so this crate takes no position on the exporter, and the emit sites compile away with the feature off.

Shape

lance_mem_wal_flush_duration_seconds{kind="wal"|"memtable"} — one family with a label rather than two, because the WAL buffer flush and the memtable flush are stages of the same write pipeline and get read together. They differ by orders of magnitude, hence bucket bounds spanning a single object-store round trip through a multi-second dataset write.

Counts and byte totals stay on WriteStats: cumulative values lose nothing to sampling, so there is no reason to route them through a recorder.

Notes

  • New dataset/mem_wal/metrics.rs, mirroring lance_io::object_store::metrics (name constants, bucket bounds, a describe_metrics an exporter calls after installing its recorder).
  • Two emit sites, in WriteStats::record_wal_flush and record_memtable_flush, where the individual duration is already in hand.
  • metrics becomes an optional dependency of the lance crate; the existing metrics feature now enables it alongside lance-io/metrics.
  • Compiles with the feature on and off; mem_wal tests pass both ways.

@github-actions github-actions Bot added A-deps Dependency updates enhancement New feature or request labels Aug 18, 2026
hamersaw and others added 3 commits August 18, 2026 20:23
`WriteStats` already tracks flush counts and cumulative time, but a running
total cannot be resampled into a distribution — the individual observations
are gone by the time anything polls it. An embedder can compute an average
and nothing else, which is exactly the wrong shape for latency: a flush
pipeline is judged on its tail, not its mean.

Observe each flush individually instead, through the `metrics` facade that
`lance-io` already uses for object store operations. Observations route to
whatever `Recorder` the embedding process installed, so this crate takes no
position on the exporter and the emit sites compile away with the feature
off.

One family with a `kind` label rather than two: a WAL buffer flush and a
memtable flush are stages of the same write pipeline and are read together,
even though they differ by orders of magnitude — hence bucket bounds
spanning a single object-store round trip through a multi-second dataset
write.

Counts and byte totals stay on `WriteStats`. They are cumulative and lose
nothing to sampling, so there is no reason to route them through a recorder.
Replaces the `metrics`-facade approach from the previous commit.

The problem is unchanged: `WriteStats` tracks flush counts and cumulative
time, but a running total cannot be resampled into a distribution. An
embedder can compute an average and nothing else, which is the wrong shape
for latency — a flush pipeline is judged on its tail, not its mean.

Report each flush to an optional `WalObserver` on `ShardWriterConfig`,
alongside `warmer`. The consumer supplies the sink and owns the aggregation,
so Lance still takes no position on the exporter, but now needs no feature
flag and no process-global recorder.

An injected sink rather than the facade because the consumer holds context
Lance does not — the table a shard belongs to, in particular, which a
process-global histogram cannot label. It also matches how consumers already
reach into this config: `SsTableWarmer` and `DatasetCache` cross the same
boundary the same way, while the facade has one producer in the tree and no
consumer that installs a recorder.

Every trait method defaults to a no-op, so adding an event later is not a
breaking change for implementors.

Counts and byte totals stay on `WriteStats`: they are cumulative and lose
nothing to sampling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The write bench builds `ShardWriterConfig` field-by-field with no
`..default()`, so adding `observer` broke every job that checks
benchmarks: clippy, MSRV, build-no-lock, and the "Check benchmarks" step
on mac and windows. Set it to `None` beside the sibling `warmer`.

Add the test the observer commit was missing. A durable put returns only
once its WAL flush landed, and the seal fence resolves only once the
sealed memtable reached L0, so both callbacks have fired by the time it
asserts — no sleeping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hamersaw
hamersaw force-pushed the feature/mem-wal-flush-metrics branch from acad3c0 to baba7a3 Compare August 19, 2026 01:23
@xuanyu-z

Copy link
Copy Markdown
Contributor

@claude review once

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants