Repository navigation
Avoid writing empty placeholder files for date partitions with no data - #13
Merged
Merged
Conversation
PartitionedSource addresses each date via BigQuery's partition decorator (table$YYYYMMDD), which returns zero rows rather than raising NotFound when a partition has no data -- unlike ShardedSource, where a missing shard table does raise NotFound. This meant non-sharded sources silently wrote a schema-only placeholder file for every empty date (e.g. before a source's real start date, or any gap day), cluttering the output with tiny files that hurt query performance on the external table. Source stays a pure addressing/naming type: dates_with_data_query() returns the check as data (a query + parameters), not by running it, so it never needs a BQ client. Exporter (which already owns the BQ client) runs the query and filters dates accordingly. There's no safe generic default for this on the base class, same as separator: a source that can't support the check must explicitly return None (a deliberate choice, not an inherited omission), which Exporter treats as "assume every date has data". ShardedSource gets the same proactive check via INFORMATION_SCHEMA.TABLES, instead of relying on catching NotFound from a submitted extract job. NotFound is a generic "resource missing" error -- it fires the same way for a missing shard, a typo'd dataset, or a missing project, so treating it as proof "this date has no data" was unreliable: a misconfigured source could make every date look like a benign gap instead of a hard failure. A truly wrong project/dataset now fails immediately anyway, since Exporter.dates_with_data() runs its check with no try/except, before any job is submitted. Since dates are now confirmed to exist before a job is ever submitted, a NotFound at job-completion time is an anomaly (e.g. a genuine race: the table dropped between the check and the job running), not a routine outcome. It's handled by the same generic except Exception path as any other unexpected failure -- no special case needed, since the outcome (fail hard, surface via run()'s RuntimeError, get retried) is identical either way. ExportStatus. NOT_FOUND and ExportJobResults.skipped are removed as a result: nothing should intentionally produce that status anymore.
tomaslink
force-pushed
the
fix/bq-to-parquet-skip-empty-partitions
branch
from
August 5, 2026 19:56
3a95f41 to
502569b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://globalfishingwatch.atlassian.net/browse/PIPELINE-4368
Avoid writing empty placeholder files for date partitions with no data
PartitionedSourcewrites a schema-only file even for empty/non-existent partitions (BigQuery'stable$YYYYMMDDdecorator returns 0 rows instead of erroring) — cluttering the external table with tiny files that hurt query performance.INFORMATION_SCHEMA.PARTITIONS/.TABLES, metadata-only, no scan) and skipping dates with no data before submitting a job.ShardedSourcegets the same proactive check, replacing its old reliance on catchingNotFound— that exception fires for any missing resource (table, dataset, project), so it couldn't reliably distinguish a missing shard from a misconfigured source.