Add timestamp and version metadata columns - #42
Conversation
This commit bumps duckdb-osmium to v0.5, which means layers are now allowed to contain NULL geometries. This is permitted by the GeoParquet spec, but may be a breaking change for some workflows. Add a predicate like `WHERE geometry IS NOT NULL` when fetching data if you need to ensure that all rows have a geometry.
|
I'm mostly just using the import scripts and running over custom regions, but I'm supportive of the NULL geometry bump. It took me quite a bit of head scratching to figure out why some rows just seemed to disappear, because the geometry couldn't be constructed. This was running the scripts over some geofabrik regions. Occasionally you'll see points that are referenced by ID which have been culled from the dataset because the points are outside of the bounds of the region. This lead to some weirdness like dropping some counties/boundary=administrative because the of differences between the region culling code and the points defining the borders of some states. It seems a lot easier to figure out what is going on when you're not dropping the whole row silently, just the geometry column. It wasn't until it happened to a large/obviously region like an administrative boundary that I figured out what was going on. |
This bumps duckdb-osmium to v0.5, which supports reading OSM PBF metadata, so we can add
version(the OSM element's version number) andtimestamp(when the element was last modified) columns to Layercake layers. These are useful signals about how likely the record is to be correct and up to date.As a side effect of bumping duckdb-osmium to v0.5, layers are now allowed to contain NULL geometries. This is permitted by the GeoParquet spec, but may be a breaking change for some workflows.
I think it's okay for Layercake to publish rows with null geometries, and expect downstream consumers to add a predicate like
WHERE geometry IS NOT NULLwhen fetching data if they need to ensure that all rows have a geometry. But if anyone has opinions on this, leave a comment and let me know.