fix(pipeline): a price change between two garbage nights was published as a market move by treyes133 · Pull Request #208 · the-tin-app/the_tin

treyes133 · 2026-10-05T04:04:26Z

What & why

Movers → Market → 1M led with Sceptile (Great Encounters, Reverse Holofoil) at +454.9%, and the card screen agreed (after #201), but its own price history moved +16.8%. The pipeline's change computation was the culprit.

A price change is the difference between two single-night PPT snapshots, and for thin markets either night can be incoherent. On Sep 4, Sceptile's Reverse Holofoil had NM $2.97 under Damaged $4.24. Against the preserved v92 (Oct 4) and its 30-day basis v62 (Sep 4) there were 512 changes past +500% (max +343,634%), and the app's +500% clamp left the band just under it on screen. The file pick (by mtime) and the pipeline image were both ruled out: the same image built both nights, and the 30d basis was exactly 30.0 days old.

computePriceDeltas now only publishes a change where both nights are coherent (computePriceDeltasFrom, documented in place):

Inversions: prices on one ladder that run backwards by more than 1.5× blame the cell caught in the most such pairs (both, on a tie). Applies to a printing's condition cells, the card-level conditions, and the PSA grades. A printing's own change and raw quote the top of their ladder, so they go when that cell does. This catches Kyogre Star (MP $299.99 < HP $600, label matched both nights) and Blaziken (PSA 10 $16 < PSA 7 $136).
Modelled printing: a printing needs ≥2 ranked conditions and the same top condition on both nights. This drops PPT category buckets posing as printings ("Miscellaneous Cards & Products": Charizard +443%, Meltan +1465%), damaged-only "market" prices, and quote switches (Rayquaza LP $112 → NM $400).
Corroboration: a move past ±50% needs another price on its own ladder moving ≥20% the same way, and at least ¼ of the headline. A real market lifts the ladder; a bad quote lifts one cell (Team Aqua's Corphish NM +378% with MP/HP/DMG byte-identical). It also catches Togepi PSA 10 at $222.50 → $150,000 on a +50% witness.
Same product: the overnight sweep now records price_latest.price_source (the PPT product's tcgPlayerId). A card whose product changed gets no change of any kind. This becomes active as artifacts carrying the column age into each window.

Rules only ever remove a change; a kept change is computed exactly as before. scripts/validate-deltas.ts replays this against preserved artifacts on a copy and checks both.

Validation on the real artifacts

Run by a peer session against the published v92 / v91 / v85 / v62 (sha256-verified against the nightly manifest), copies only.

Changes kept (dropped %):

	1d	7d	30d
condition	93,766 → 85,858 (8%)	93,661 → 84,853 (9%)	92,908 → 82,590 (11%)
matrix	150,767 → 138,778 (8%)	150,582 → 137,233 (9%)	149,571 → 133,421 (11%)
printing	37,936 → 32,734 (14%)	37,900 → 32,389 (15%)	37,696 → 31,824 (16%)
raw	21,147 → 19,961 (6%)	21,133 → 19,686 (7%)	20,913 → 19,296 (8%)
psa	35,775 → 31,927 (11%)	35,584 → 31,146 (12%)	33,559 → 27,878 (17%)

Past ±500%, 30d: condition 81 → 0, matrix 148 → 0, printing 26 → 0, raw 7 → 0, psa 256 → 28. Condition, matrix, printing and raw are at zero in every window.
Kept rows whose value changed: 0. Rows that appeared from nothing: 0, in all three windows (also re-derived independently).
The reported movers are all gone: Sceptile, Loudred, Marill, Charizard, Kyogre Star, plus Meltan, Blaziken, Togepi, Corphish.
The new Market 1M list opens with Zoroark +292%, a verified whole-ladder move (LP +57%, MP +93%, DMG +423%), and tapers to +72.5%.

Earlier iterations on this branch are kept for the record: a strict "any inversion" rule dropped ~60% of every window, and a ≥3-sales graded gate cost ~40% of psa for little gain. Both were replaced.

Known, not in this PR

28 psa rows past ±500% remain from implausibly steep grade ladders (correct order, absurd gap: Leafeon PSA 10 at 49× its PSA 9). These don't reach Movers → Market (which is printing/raw only), but show on graded tiles. A cap on the adjacent-grade ratio would be the follow-up.
Odd-one-out can blame the wrong side when a ladder has two anomalies (ex3-100 Charizard). Rare.
Real print runs without a condition breakdown (Cosmos Holo, Cracked Ice Holo, Prize Pack) lose their changes. That's ~1,100 rows, mostly small; an allow-list could restore them. The same category buckets also still appear in the card screen's printing menu, which is a separate catalog-quality fix.

Deploying

No app release is needed; every installed app gets correct changes from the next nightly after the pipeline image is rebuilt. price_source is a new nullable column that the app ignores (SELECT *, read by name).

Checklist

Tests pass locally: functions vitest, 353 passed / 1 skipped. 15 new sanity-rule tests built from the real cases, each failing with its rule switched off. On macOS use --pool=forks: better-sqlite3 segfaults at teardown under worker threads, and unmodified staging does the same.
No secrets, .env, keys, or docs/ contents committed
Targets staging

🤖 Generated with Claude Code

…d as a market move Movers → Market → 1M led with Sceptile (Great Encounters Reverse Holofoil) at +454.9% while its own price history moved +16.8%. Measured against the preserved v92 (2026-10-04) and its 30d basis v62 (2026-09-04): 512 deltas past +500% (max +343,634%), and the app's +500% clamp left the band just under it on screen. Same pipeline image for both nights, mtime pick exact — the cause is the data. A delta is two single-night PPT snapshots, and for thin markets either night can be incoherent: on Sep 4 Sceptile's Reverse Holofoil had NM $2.97 under Damaged $4.24. computePriceDeltas now only writes a delta when both nights describe something coherent: 1. Ladder — a printing whose condition prices run backwards (a better condition below a worse one) on either night gets no printing/matrix delta, and no raw delta when raw quotes it. Kyogre Star kept raw_printing 'Holofoil' both nights, so the label guard passed its +434.3%; its Sep 4 ladder (MP $299.99 < HP $600) doesn't. The card-level ladder gates `condition`. 2. No ladder — a price_by_variant row with no price_matrix rows is not a printing this pipeline models ("Miscellaneous Cards & Products", "League & Championship Cards": 3,327 rows, 8.8% of printing deltas, carrying Charizard's +443% and Meltan's +1465%). 3. Thin graded — a psa delta needs ≥3 sales behind the grade tonight. 4. Same product — the sweep now records price_latest.price_source (the PPT product's tcgPlayerId), and a card whose product changed gets no delta at all. Artifacts from before the column skip this check. Each rule applies only where both databases carry what it reads, so older artifacts keep their behaviour (the existing 23 tests pass unchanged). The diff is split out as computePriceDeltasFrom so scripts/validate-deltas.ts can replay it against preserved artifacts and report what each window loses — validation on copies, never the served catalog dir. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e bad pair, not the ladder Validation of 672b059 against the preserved v92/v91/v85/v62 artifacts: zero kept rows altered, all five reported movers gone — and ~60–65% of every kind's deltas dropped, in every window (1d raw 21,147 → 8,921). Any inversion anywhere disqualified the whole printing, and thin markets invert by cents all the time (Damaged a little over Heavily Played). Charizard's real Normal printing (+0.3%) and every Loudred row went with the garbage. Four defects also survived: - Grade ladders were never checked: Blaziken's PSA 10 was $16 under PSA 7 $136 on Sep 4, and its +343,634% passed (it had 9 sales). - A one-condition ladder passed vacuously: a Damaged-only Reverse Holofoil read as the market, +275% (Market #4). - The variant price's condition basis could change: Rayquaza's Normal quoted LP $112.06 one night and NM $400 the next, +257% with every cell unchanged (Market #5). Now: 1. Only an inversion past INVERSION_FACTOR (1.5×) counts, and it blanks the two prices in that pair — on printing ladders, card-level conditions, and now PSA grades. A printing's own delta and raw quote the top of their ladder, so they go when that top cell does (Kyogre Star, Sceptile, Loudred, Marill all invert by 1.9–6.4×; Blaziken by 8.5×). 2. A printing delta needs ≥2 ranked conditions and the same top condition on both nights; raw gets the same-top check where its printing has a ladder on both nights. Tests pin each rule and the targeting (a bad pair low on the ladder leaves the printing alone); each fails with its rule switched off. validate-deltas.ts notes the ~1.3 GB-per-.gz heap that OOM-killed the first run on a small host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ket list Second validation of the sanity rules (eaaabbd vs preserved v92/v91/v85/v62): condition, matrix, printing and raw back to 9–18% dropped, every original case fixed, zero kept rows altered. Two things left: - The Market 1M top 25 was mostly single-cell Near Mint jumps on ladders whose other cells sat still: Team Aqua's Corphish Reverse Holofoil NM $10.45 → $49.99 (+378%) with MP/HP/DMG byte-identical a month apart. 15 of the 25 had no other cell move even 20%. - psa lost 52%: the ≥3-sales gate cost ~40% on its own (and still passed Blaziken, at 11 sales), and blaming both sides of every inverted pair let one $16 PSA 10 blank the whole grade ladder. So: - Corroboration (new rule 3): a move past ±50% survives only if another price on its own ladder — another condition of the printing, another grade — moved ≥20% the same way. A real market lifts more than one cell; a bad quote lifts only the one quoted. A printing and raw are witnessed by their printing's OTHER conditions, never the cell they quote. Witnesses are judged on the rows rules 1/2/4 kept, all decided before any is cleared. - Inversions blame only the cell caught in the most pairs (both, on a tie, when nothing says which is wrong): the $16 PSA 10 goes alone and PSA 7 keeps its -17.5%; Kyogre's MP goes, its HP/DMG stay. - The ≥3-sales graded gate is gone; corroboration does that job without the coverage cost. Each rule is pinned by a test that fails with it switched off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Third validation (6015be1): every kind at 6–16% dropped, printing and raw clean of >500% rows in every window, zero kept rows altered — but corroboration tested a witness's presence, not its proportion. A Togepi PSA 10 at $222.50 → $150,000 (+67,316%) stood on a PSA 8 +50%; 13 of the Market top 25 led their best witness by more than 4× (Zapdos +318.9% on +28.9%). The witness must now also be at least 1/4 of the headline. Whole-ladder moves are untouched by it (Zoroark +292% on a DMG +423%; Blaziken PSA 8/9 vouching for each other at ~4× each). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

### fix(pipeline): a price change between two garbage nights was published as a market move Movers → Market → 1M led with Sceptile (Great Encounters Reverse Holofoil) at +454.9% while its own price history moved +16.8%. Measured against the preserved v92 (2026-10-04) and its 30d basis v62 (2026-09-04): 512 deltas past +500% (max +343,634%), and the app's +500% clamp left the band just under it on screen. Same pipeline image for both nights, mtime pick exact — the cause is the data. A delta is two single-night PPT snapshots, and for thin markets either night can be incoherent: on Sep 4 Sceptile's Reverse Holofoil had NM $2.97 under Damaged $4.24. computePriceDeltas now only writes a delta when both nights describe something coherent: 1. Ladder — a printing whose condition prices run backwards (a better condition below a worse one) on either night gets no printing/matrix delta, and no raw delta when raw quotes it. Kyogre Star kept raw_printing 'Holofoil' both nights, so the label guard passed its +434.3%; its Sep 4 ladder (MP $299.99 < HP $600) doesn't. The card-level ladder gates `condition`. 2. No ladder — a price_by_variant row with no price_matrix rows is not a printing this pipeline models ("Miscellaneous Cards & Products", "League & Championship Cards": 3,327 rows, 8.8% of printing deltas, carrying Charizard's +443% and Meltan's +1465%). 3. Thin graded — a psa delta needs ≥3 sales behind the grade tonight. 4. Same product — the sweep now records price_latest.price_source (the PPT product's tcgPlayerId), and a card whose product changed gets no delta at all. Artifacts from before the column skip this check. Each rule applies only where both databases carry what it reads, so older artifacts keep their behaviour (the existing 23 tests pass unchanged). The diff is split out as computePriceDeltasFrom so scripts/validate-deltas.ts can replay it against preserved artifacts and report what each window loses — validation on copies, never the served catalog dir. ### fix(pipeline): the ladder rule removed 60% of every window — blank the bad pair, not the ladder Validation of 672b059 against the preserved v92/v91/v85/v62 artifacts: zero kept rows altered, all five reported movers gone — and ~60–65% of every kind's deltas dropped, in every window (1d raw 21,147 → 8,921). Any inversion anywhere disqualified the whole printing, and thin markets invert by cents all the time (Damaged a little over Heavily Played). Charizard's real Normal printing (+0.3%) and every Loudred row went with the garbage. Four defects also survived: - Grade ladders were never checked: Blaziken's PSA 10 was $16 under PSA 7 $136 on Sep 4, and its +343,634% passed (it had 9 sales). - A one-condition ladder passed vacuously: a Damaged-only Reverse Holofoil read as the market, +275% (Market #4). - The variant price's condition basis could change: Rayquaza's Normal quoted LP $112.06 one night and NM $400 the next, +257% with every cell unchanged (Market #5). Now: 1. Only an inversion past INVERSION_FACTOR (1.5×) counts, and it blanks the two prices in that pair — on printing ladders, card-level conditions, and now PSA grades. A printing's own delta and raw quote the top of their ladder, so they go when that top cell does (Kyogre Star, Sceptile, Loudred, Marill all invert by 1.9–6.4×; Blaziken by 8.5×). 2. A printing delta needs ≥2 ranked conditions and the same top condition on both nights; raw gets the same-top check where its printing has a ladder on both nights. Tests pin each rule and the targeting (a bad pair low on the ladder leaves the printing alone); each fails with its rule switched off. validate-deltas.ts notes the ~1.3 GB-per-.gz heap that OOM-killed the first run on a small host. ### fix(pipeline): a lone quote jump on a frozen ladder still led the Market list Second validation of the sanity rules (eaaabbd vs preserved v92/v91/v85/v62): condition, matrix, printing and raw back to 9–18% dropped, every original case fixed, zero kept rows altered. Two things left: - The Market 1M top 25 was mostly single-cell Near Mint jumps on ladders whose other cells sat still: Team Aqua's Corphish Reverse Holofoil NM $10.45 → $49.99 (+378%) with MP/HP/DMG byte-identical a month apart. 15 of the 25 had no other cell move even 20%. - psa lost 52%: the ≥3-sales gate cost ~40% on its own (and still passed Blaziken, at 11 sales), and blaming both sides of every inverted pair let one $16 PSA 10 blank the whole grade ladder. So: - Corroboration (new rule 3): a move past ±50% survives only if another price on its own ladder — another condition of the printing, another grade — moved ≥20% the same way. A real market lifts more than one cell; a bad quote lifts only the one quoted. A printing and raw are witnessed by their printing's OTHER conditions, never the cell they quote. Witnesses are judged on the rows rules 1/2/4 kept, all decided before any is cleared. - Inversions blame only the cell caught in the most pairs (both, on a tie, when nothing says which is wrong): the $16 PSA 10 goes alone and PSA 7 keeps its -17.5%; Kyogre's MP goes, its HP/DMG stay. - The ≥3-sales graded gate is gone; corroboration does that job without the coverage cost. Each rule is pinned by a test that fails with it switched off. ### fix(pipeline): a witness licensed a headline of any size Third validation (6015be1): every kind at 6–16% dropped, printing and raw clean of >500% rows in every window, zero kept rows altered — but corroboration tested a witness's presence, not its proportion. A Togepi PSA 10 at $222.50 → $150,000 (+67,316%) stood on a PSA 8 +50%; 13 of the Market top 25 led their best witness by more than 4× (Zapdos +318.9% on +28.9%). The witness must now also be at least 1/4 of the headline. Whole-ladder moves are untouched by it (Zoroark +292% on a DMG +423%; Blaziken PSA 8/9 vouching for each other at ~4× each). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Tomas Reyes and others added 4 commits October 4, 2026 21:20

treyes133 merged commit 5461d40 into staging Oct 5, 2026
5 checks passed

treyes133 mentioned this pull request Oct 5, 2026

Promote to main: price-delta sanity rules (cherry-pick of #208) #209

Merged

Navigation Menu

Repository navigation

fix(pipeline): a price change between two garbage nights was published as a market move - #208

fix(pipeline): a price change between two garbage nights was published as a market move#208
treyes133 merged 4 commits into
stagingfrom
fix/price-delta-sanity

treyes133 commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

treyes133 commented Oct 5, 2026

What & why

Validation on the real artifacts

Known, not in this PR

Deploying

Checklist

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant