Add mobile_verifier_compare crate: PG ↔ Trino reward-query parity CLI#1199
Open
macpie wants to merge 4 commits into
Open
Add mobile_verifier_compare crate: PG ↔ Trino reward-query parity CLI#1199macpie wants to merge 4 commits into
macpie wants to merge 4 commits into
Conversation
New standalone binary crate that diffs every Postgres reward-input query mobile_verifier runs against an equivalent query over the matching Iceberg tables in Trino (poc.heartbeats, poc.speedtests, rewards.data_transfer), then prints a side-by-side diff with per-row MATCH / MISMATCH / PG_ONLY / TRINO_ONLY status. The Postgres side reuses the production functions (HeartbeatReward::validated, aggregate_epoch_speedtests, aggregate_hotspot_data_sessions_to_dc, sum_data_sessions_to_dc_by_payer) via a path-dep on mobile_verifier so the SQL cannot drift from what the reward pipeline actually executes. Ships a `seed` subcommand that bootstraps the local docker-compose stack end-to-end: creates the mobile_verifier and mobile_packet_verifier Postgres databases, runs sqlx migrations against both, creates the Iceberg schemas + tables via Trino DDL, and populates deterministic fixtures across four buckets so every diff status appears on a first run with zero prior config. Lives in its own workspace member so mobile_verifier release builds contain no Trino client code — the prod/dev boundary is enforced by the dep graph rather than a Cargo feature flag.
bbalser
reviewed
Jun 8, 2026
…coding mobile_verifier_compare hardcoded SPEEDTEST_LAPSE_HOURS to 24, but the production constant is 48 — speedtest comparisons were diffing against the wrong sliding window. SPEEDTEST_LAPSE was already pub, so the fix is a direct import. Audited the rest of the parity-critical constants and found two more that were copied by value (and could drift the same way): - MINIMUM_HEARTBEAT_COUNT (12) was hardcoded in both the Trino HAVING clause and the seed fixture counts. - SPEEDTEST_AVG_MAX_DATA_POINTS (6) was hardcoded in the Trino window function bind, the Trino LIMIT clause, and the seed row count. Bumped both to pub in mobile_verifier and imported them. The Trino HAVING uses :min_count bind; the LIMIT uses format! since Trino doesn't allow LIMIT placeholders. Seed fixtures derive their counts from the imported constants so future production-side tuning automatically propagates.
bbalser
reviewed
Jun 12, 2026
bbalser
reviewed
Jun 12, 2026
Two bugs spotted by @bbalser on #1199 that would have produced spurious MISMATCH rows for cases where production and Iceberg actually agree: heartbeats: the Trino CTE averaged location_trust_score_multiplier across every heartbeat in an hour, but production picks one value per hour — the multiplier of the first heartbeat to land in that hour. PG enforces this via wifi_heartbeats's (hotspot_key, truncated_timestamp) PK plus an ON CONFLICT DO UPDATE clause that only refreshes first_timestamp + coverage_object, leaving the trust multiplier sticky. Switched the CTE to min_by(location_trust_score_multiplier, heartbeat_timestamp) so the Trino side mirrors that semantics exactly. data_sessions: the comparison diffed PG's `num_dcs` against Trino's `dc_transfer_reward`. Those are different units — num_dcs is the count of DCs the session consumes (input to the burn step) while dc_transfer_reward is the paid reward in bones (output of burn × price). Dropped that column from the diff entirely; only rewardable_bytes is apples-to-apples across the two tables. Rewrote the module doc to explain what is and isn't being validated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New standalone binary crate
mobile-verifier-comparethat diffs every Postgres reward-input querymobile_verifierruns against an equivalent query over the matching Iceberg tables in Trino (poc.heartbeats,poc.speedtests,rewards.data_transfer), then prints a side-by-side table with per-rowMATCH/MISMATCH/PG_ONLY/TRINO_ONLYstatus.Lives in its own workspace member —
mobile_verifieritself is untouched. The prod/dev boundary is enforced by the dep graph rather than a Cargo feature flag, so release builds ofmobile_verifiercontain no Trino client code by construction.Why
We want to verify that the Iceberg-side data the new backfillers and writers produce is byte-equivalent to what the Postgres-side reward pipeline reads, before we trust downstream consumers to use Iceberg instead of PG. This is the validation harness for that work.
What's in the crate
Diff subcommands
Each subcommand calls the production PG function so the Postgres SQL cannot drift from what the reward pipeline actually executes:
heartbeatsHeartbeatReward::validated(valid_radios.sql)poc.heartbeatswithcount(*) >= 12filterspeedtestsaggregate_epoch_speedtestspoc.speedtestslatest-speedtestsget_latest_speedtests_for_pubkeydata-sessionsaggregate_hotspot_data_sessions_to_dcrewards.data_transferover the windowpayer-burnssum_data_sessions_to_dc_by_payerrewards.data_transfer(header notes Iceberg has no payer column)pending-burnsmobile_packet_verifier::pending_burns::initializeSQLallThe diff table shows mismatches by default (
--show-allto dump matches too) and prints totals + max delta.seedsubcommand (zero-config bootstrap)Brings the local docker-compose stack from nothing to a runnable diff in one command:
mobile_verifierandmobile_packet_verifierPostgres databases on the docker-compose PG.sqlxmigrations against both (paths resolved relative toCARGO_MANIFEST_DIR).CREATE SCHEMA+CREATE TABLEforiceberg.poc.heartbeats,iceberg.poc.speedtests,iceberg.rewards.data_transfervia Trino DDL.mobile_verifier_compare/pkg/compare-trino-settings.local.toml(gitignored) with docker-compose defaults, and prints the exact nextcargo run … allcommand to copy.Takes every endpoint as a CLI flag with docker-compose-friendly defaults — works with no
-cconfig file.Reviewer notes
rewards.data_transferoutput rewards — the printed table headers call out that this is an input-vs-output sanity check, not strict row equality. A future raw-sessions Iceberg writer would let us swap that SQL.payer-burnsrolls Iceberg to a single__all__bucket becauserewards.data_transferhas no payer column. Per-payer PG rows are shown for visibility but always tag asPG_ONLY— by design today.pending-burnsis one-sided (no Iceberg counterpart for unprocessed sessions); reported asPG_ONLYwith a header note.