Repository navigation
Promote to main: price-delta sanity rules (cherry-pick of #208) - #209
Merged
Merged
Conversation
…d as a market move (#208) ### fix(pipeline): a price change between two garbage nights was published as a market move Movers → Market → 1M led with Sceptile (Great Encounters Reverse Holofoil) at +454.9% while its own price history moved +16.8%. Measured against the preserved v92 (2026-10-04) and its 30d basis v62 (2026-09-04): 512 deltas past +500% (max +343,634%), and the app's +500% clamp left the band just under it on screen. Same pipeline image for both nights, mtime pick exact — the cause is the data. A delta is two single-night PPT snapshots, and for thin markets either night can be incoherent: on Sep 4 Sceptile's Reverse Holofoil had NM $2.97 under Damaged $4.24. computePriceDeltas now only writes a delta when both nights describe something coherent: 1. Ladder — a printing whose condition prices run backwards (a better condition below a worse one) on either night gets no printing/matrix delta, and no raw delta when raw quotes it. Kyogre Star kept raw_printing 'Holofoil' both nights, so the label guard passed its +434.3%; its Sep 4 ladder (MP $299.99 < HP $600) doesn't. The card-level ladder gates `condition`. 2. No ladder — a price_by_variant row with no price_matrix rows is not a printing this pipeline models ("Miscellaneous Cards & Products", "League & Championship Cards": 3,327 rows, 8.8% of printing deltas, carrying Charizard's +443% and Meltan's +1465%). 3. Thin graded — a psa delta needs ≥3 sales behind the grade tonight. 4. Same product — the sweep now records price_latest.price_source (the PPT product's tcgPlayerId), and a card whose product changed gets no delta at all. Artifacts from before the column skip this check. Each rule applies only where both databases carry what it reads, so older artifacts keep their behaviour (the existing 23 tests pass unchanged). The diff is split out as computePriceDeltasFrom so scripts/validate-deltas.ts can replay it against preserved artifacts and report what each window loses — validation on copies, never the served catalog dir. ### fix(pipeline): the ladder rule removed 60% of every window — blank the bad pair, not the ladder Validation of 672b059 against the preserved v92/v91/v85/v62 artifacts: zero kept rows altered, all five reported movers gone — and ~60–65% of every kind's deltas dropped, in every window (1d raw 21,147 → 8,921). Any inversion anywhere disqualified the whole printing, and thin markets invert by cents all the time (Damaged a little over Heavily Played). Charizard's real Normal printing (+0.3%) and every Loudred row went with the garbage. Four defects also survived: - Grade ladders were never checked: Blaziken's PSA 10 was $16 under PSA 7 $136 on Sep 4, and its +343,634% passed (it had 9 sales). - A one-condition ladder passed vacuously: a Damaged-only Reverse Holofoil read as the market, +275% (Market #4). - The variant price's condition basis could change: Rayquaza's Normal quoted LP $112.06 one night and NM $400 the next, +257% with every cell unchanged (Market #5). Now: 1. Only an inversion past INVERSION_FACTOR (1.5×) counts, and it blanks the two prices in that pair — on printing ladders, card-level conditions, and now PSA grades. A printing's own delta and raw quote the top of their ladder, so they go when that top cell does (Kyogre Star, Sceptile, Loudred, Marill all invert by 1.9–6.4×; Blaziken by 8.5×). 2. A printing delta needs ≥2 ranked conditions and the same top condition on both nights; raw gets the same-top check where its printing has a ladder on both nights. Tests pin each rule and the targeting (a bad pair low on the ladder leaves the printing alone); each fails with its rule switched off. validate-deltas.ts notes the ~1.3 GB-per-.gz heap that OOM-killed the first run on a small host. ### fix(pipeline): a lone quote jump on a frozen ladder still led the Market list Second validation of the sanity rules (eaaabbd vs preserved v92/v91/v85/v62): condition, matrix, printing and raw back to 9–18% dropped, every original case fixed, zero kept rows altered. Two things left: - The Market 1M top 25 was mostly single-cell Near Mint jumps on ladders whose other cells sat still: Team Aqua's Corphish Reverse Holofoil NM $10.45 → $49.99 (+378%) with MP/HP/DMG byte-identical a month apart. 15 of the 25 had no other cell move even 20%. - psa lost 52%: the ≥3-sales gate cost ~40% on its own (and still passed Blaziken, at 11 sales), and blaming both sides of every inverted pair let one $16 PSA 10 blank the whole grade ladder. So: - Corroboration (new rule 3): a move past ±50% survives only if another price on its own ladder — another condition of the printing, another grade — moved ≥20% the same way. A real market lifts more than one cell; a bad quote lifts only the one quoted. A printing and raw are witnessed by their printing's OTHER conditions, never the cell they quote. Witnesses are judged on the rows rules 1/2/4 kept, all decided before any is cleared. - Inversions blame only the cell caught in the most pairs (both, on a tie, when nothing says which is wrong): the $16 PSA 10 goes alone and PSA 7 keeps its -17.5%; Kyogre's MP goes, its HP/DMG stay. - The ≥3-sales graded gate is gone; corroboration does that job without the coverage cost. Each rule is pinned by a test that fails with it switched off. ### fix(pipeline): a witness licensed a headline of any size Third validation (6015be1): every kind at 6–16% dropped, printing and raw clean of >500% rows in every window, zero kept rows altered — but corroboration tested a witness's presence, not its proportion. A Togepi PSA 10 at $222.50 → $150,000 (+67,316%) stood on a PSA 8 +50%; 13 of the Market top 25 led their best witness by more than 4× (Zapdos +318.9% on +28.9%). The witness must now also be at least 1/4 of the headline. Whole-ladder moves are untouched by it (Zoroark +292% on a DMG +423%; Blaziken PSA 8/9 vouching for each other at ~4× each). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #208 straight to
main, the same shape as #186 → #185 and #173 → #172.The catalog pipeline runs on the NAS from an image built off
main; it doesn't ship inside the app. Without this, a server-side price fix waits on the 1.0.6 app release, which has nothing to do with it. Meanwhile the live catalog keeps serving changes like Sceptile Reverse Holofoil +454.9% (v93, published 2026-10-05, still has 26 printing / 147 matrix / 266 psa changes past +500% at 30d).Identical commit (
5461d40onstaging). The four files it touches were byte-identical betweenmainandstagingbefore it, so it applies cleanly. Verified againstmain(which also carries #190):tsc --noEmitclean, 359 passed, 1 skipped, 0 failures.What it fixes
The nightly now publishes a price change only where both nights are coherent:
price_latest.price_sourcecolumn blocks diffs across a PPT re-match.Validated against the published v92/v91/v85/v62 artifacts, copies only:
Full numbers are in #208.
Deploy
After merge, rebuild
catalog-pipeline:latestfrommain, keepingrollback-2026-10-05for the Aug 29 image. The next nightly corrects every installed app.price_sourceis a new nullable column that the app ignores.🤖 Generated with Claude Code