Skip to content

Fix positions_hours_before undercounted for open gaps at window overlaps - #146

Merged
tomaslink merged 2 commits into
mainfrom
fix-boundary-stitching-truncated-lookback
Oct 2, 2026
Merged

tomaslink merged 2 commits into
mainfrom
fix-boundary-stitching-truncated-lookback

Conversation

@tomaslink

Copy link
Copy Markdown
Collaborator

Summary

  • Boundaries.last_boundary() picked whichever boundary starts latest, regardless of whether it had the fullest history for the vessel's actual last message. When that message falls in the ~12h overlap between two sliding windows, the later-starting window's own lookback is shorter, so the open gap created in Step three (eval_last) got a truncated positions_hours_before/positions_hours_before_sat.
  • Added Boundaries.boundary_for_last_message(), which picks the earliest-starting boundary whose own last message matches the overall last message across all boundaries — i.e. the boundary with the fullest lookback, not just whichever window starts latest.
  • Confirmed against real production data (gfw-int-ais-datalake.gaps_v1.raw_gaps_versioned): this recurs at every internal sliding-window boundary inside both the daily-incremental DAG and the yearly backfill runs (window period caps at 180 days per MAX_WINDOW_PERIOD_D), with undercounting sharply concentrated in the ~12h window right after each boundary and clean everywhere else.
  • Ref: PIPELINE-4631

Test plan

  • New parametrized case one_ssvid_with_open_gap_at_window_overlap_boundary added to tests/conftest.py's TestCases.GAP_BETWEEN_DAYS, exercised by the existing test_detect_gaps.py suite
  • Full test suite passes (114 passed)
  • Verified against real BigQuery data: positions_hours_before_sat corrected from 3 to 16 for the reproduced case

Boundaries.last_boundary() picked whichever boundary starts latest, not
whichever has the fullest lookback for the vessel's last message. When that
message falls in the overlap of two sliding windows, the later-starting
window's own history is shorter, truncating positions_hours_before(_sat) for
the resulting open gap.
@tomaslink tomaslink self-assigned this Oct 2, 2026
@tomaslink
tomaslink requested a review from andres-arana October 2, 2026 19:12
@tomaslink
tomaslink merged commit 4990e36 into main Oct 2, 2026
3 checks passed
@tomaslink
tomaslink deleted the fix-boundary-stitching-truncated-lookback branch October 2, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant