perf(fts): delay must-not probes until candidates survive

[BubbleCal]

What is the performance issue?

Same-column compound FTS evaluated Boolean MUST_NOT clauses while advancing the positive driver. Candidates rejected by positive two-phase confirmation, block bounds, or the exact competitive score could still pay for prohibited posting advances and phrase confirmation. Prohibited clauses also used scoring cursors even though exclusion only needs exact membership, and their postings were loaded before the query knew whether a partition contained a competitive positive candidate.

Closes OSS-1705.

How does this PR improve performance?

This follows the same separation used by Lucene's ReqExclScorer and COMPLETE_NO_SCORES handling in BooleanScorerSupplier:

• Delay MUST_NOT confirmation until the exact positive side survives two-phase matching and the current competitive score floor. The current-document positive score and prohibited decision are cached.
• Propagate CompleteNoScores through the complete prohibited subtree, including nested Boolean, MultiMatch, and Boost queries. Membership-irrelevant scoring branches are removed while query validation remains unchanged.
• Use dedicated membership posting cursors. Non-Phrase membership decodes document IDs without frequencies, document lengths, BM25, or impact scoring. Phrase membership retains frequencies only because they delimit the shared position stream and still performs exact position confirmation.
• Load scoring leaves first. When prohibited postings are not already cached, an exact positive gate determines whether a partition can compete before loading its prohibited leaves. Deferred loads are batched concurrently, and full collection resumes at the first competitive document instead of replaying the rejected prefix.
• Use a conservative membership-only gate when signed Boost subtrees cannot prove a safe score upper envelope. Score-floor ties remain eligible, so ordered top-k semantics are unchanged.
• Reuse cached prohibited postings directly. Fully prewarmed workloads avoid the gate/replay scheduling path while still using delayed probes and no-score cursors.
• Preserve the original fast path for Boolean queries without MUST_NOT.

The query API and persisted index format do not change.

Measurement

10M-row end-to-end benchmark

Every value below is the geometric mean of four isolated process samples per build and case. Throughput is higher-is-better; latency is lower-is-better.

Scenario / metric	Baseline	This PR	Benefit
Reversed Phrase MUST_NOT, k=10 throughput	1,753.43 queries/s	1,799.34 queries/s	1.026x throughput
Reversed Phrase MUST_NOT, k=10 p99 latency	8.19 ms	7.37 ms	1.112x speedup
Reversed Phrase MUST_NOT, k=100 throughput	1,594.99 queries/s	1,603.53 queries/s	1.005x throughput
Reversed Phrase MUST_NOT, k=100 p99 latency	9.78 ms	9.16 ms	1.067x speedup
Nested doc-only MUST_NOT, k=10 throughput	1,700.42 queries/s	1,739.82 queries/s	1.023x throughput
Nested doc-only MUST_NOT, k=10 p99 latency	8.54 ms	7.85 ms	1.089x speedup
Nested doc-only MUST_NOT, k=100 throughput	1,541.75 queries/s	1,555.20 queries/s	1.009x throughput
Nested doc-only MUST_NOT, k=100 p99 latency	9.87 ms	9.73 ms	1.014x speedup
Paired no-MUST_NOT control, k=10 throughput	2,081.67 queries/s	2,072.02 queries/s	1.005x slower
Paired no-MUST_NOT control, k=10 p99 latency	6.32 ms	6.31 ms	1.002x speedup
Paired no-MUST_NOT control, k=100 throughput	1,785.00 queries/s	1,779.28 queries/s	1.003x slower
Paired no-MUST_NOT control, k=100 p99 latency	7.60 ms	7.61 ms	1.001x slower
Dense all-match MUST_NOT, k=10 throughput	2,019.56 queries/s	2,021.31 queries/s	1.001x throughput
Dense all-match MUST_NOT, k=10 p99 latency	6.68 ms	6.56 ms	1.018x speedup
Dense all-match MUST_NOT, k=100 throughput	2,035.74 queries/s	2,009.28 queries/s	1.013x slower
Dense all-match MUST_NOT, k=100 p99 latency	6.61 ms	6.63 ms	1.002x slower

The intended tight-top-k workloads improve most: k=10 gains 1.023-1.026x throughput and 1.089-1.112x p99 latency. At k=100, fewer candidates can be eliminated before exclusion, so throughput is close to flat. The dense all-match k=100 case is the expected adverse control: it never establishes a useful accepted-result floor and regresses throughput by 1.013x. The no-MUST_NOT control is within 0.5% of baseline.

Methodology: GCP c4-highmem-16 in us-central1-c (16 vCPU, 121 GiB OS-visible RAM), frozen 10,000,000-row MMLB code dataset spanning 42 languages and 10 fragments, 37.8 GiB FTS index, 64 GiB Lance index cache, 8 workers, and release-with-debug. Each case ran 1,000 deterministic queries repeated 20 times (20,000 timed executions). Two A-B-B-A blocks compared baseline 706b941b6 with target source tree 0c4bad883, reversed case order, and prewarmed positions in every process. The suite completed 1,280,000 timed executions. All 56,000 cross-process result signatures matched exactly, and the input index tree was unchanged before and after the run.

This hot-cache benchmark primarily measures delayed confirmation and no-score CPU work. The doc-only decoder reduces decoding CPU, not persisted bytes for a posting that is loaded. True cold-I/O avoidance is covered deterministically by the deferred-posting-load integration tests rather than claimed from this prewarmed latency run.

Validation

• cargo test -p lance-index --lib (1,050 passed, 2 ignored)
• Dataset compound FTS integration suite (11 passed), including deferred posting loads, visibility, Phrase false-positive rejection, and multi-fragment exactness
• Baseline and target bounded-vs-unbounded exact oracles: 24/24 cases per build, zero mismatches
• Baseline-vs-target oracle comparison: zero mismatches
• Full timed ABBA validation: 64 raw result files, 56,000 cross-process signatures, zero mismatches
• cargo check -p lance-index --tests
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings

Regression coverage includes required conjunction, required-plus-optional, pure SHOULD, nested Boolean/MultiMatch/Boost, Phrase prohibitions, multiple fragments and partitions, score-floor ties, signed negative subtrees, visibility masks, projection overflow retry, cached and uncached prohibited postings, and doc-only plain/compressed OR and AND cursors.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The positive-preflight-and-replay design defers prohibited posting I/O only for partitions that can still contribute to global top-k, while exact membership, phrase confirmation, visibility, score-floor equality, and row-ID tie ordering remain intact.

Compared with filtering a provisional top-k, this keeps the exact post-exclusion scorer authoritative, so excluded candidates never raise the shared floor.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[BubbleCal]

Closing this optimization path after the 10M MMLB results. The exact implementation improved target k=10 throughput by only 1.023-1.026x, was effectively flat at k=100, and regressed the dense all-match k=100 control by 1.013x. Rebasing and reviewing the conflicting change is not justified relative to the higher-impact follow-ups now tracked in Linear.