Skip to content

Add timestamp and version metadata columns - #42

Open
jake-low wants to merge 1 commit into
mainfrom
jlow/timestamp-and-version
Open

Add timestamp and version metadata columns#42
jake-low wants to merge 1 commit into
mainfrom
jlow/timestamp-and-version

Conversation

@jake-low

Copy link
Copy Markdown
Member

This bumps duckdb-osmium to v0.5, which supports reading OSM PBF metadata, so we can add version (the OSM element's version number) and timestamp (when the element was last modified) columns to Layercake layers. These are useful signals about how likely the record is to be correct and up to date.

As a side effect of bumping duckdb-osmium to v0.5, layers are now allowed to contain NULL geometries. This is permitted by the GeoParquet spec, but may be a breaking change for some workflows.

I think it's okay for Layercake to publish rows with null geometries, and expect downstream consumers to add a predicate like WHERE geometry IS NOT NULL when fetching data if they need to ensure that all rows have a geometry. But if anyone has opinions on this, leave a comment and let me know.

This commit bumps duckdb-osmium to v0.5, which means layers are now
allowed to contain NULL geometries. This is permitted by the GeoParquet
spec, but may be a breaking change for some workflows. Add a predicate
like `WHERE geometry IS NOT NULL` when fetching data if you need to
ensure that all rows have a geometry.
@ratmice

ratmice commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

I'm mostly just using the import scripts and running over custom regions, but I'm supportive of the NULL geometry bump.

It took me quite a bit of head scratching to figure out why some rows just seemed to disappear, because the geometry couldn't be constructed. This was running the scripts over some geofabrik regions. Occasionally you'll see points that are referenced by ID which have been culled from the dataset because the points are outside of the bounds of the region.

This lead to some weirdness like dropping some counties/boundary=administrative because the of differences between the region culling code and the points defining the borders of some states.

It seems a lot easier to figure out what is going on when you're not dropping the whole row silently, just the geometry column. It wasn't until it happened to a large/obviously region like an administrative boundary that I figured out what was going on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants