Autogenerate SCHEMA.md

.github/workflows/schema-check.yaml

@@ -0,0 +1,22 @@
+name: Check Generated Schema
+
+on:
+  push:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  schema:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Generate schema
+        run: ./schema.sh
+
+      - name: Check SCHEMA.md is up to date
+        run: git diff --exit-code -- SCHEMA.md

CONTRIBUTING.md

@@ -45,6 +45,10 @@ To add a new layer, create a new SQL file in the `sql/` directory (see existing
 
 Feel free to request new columns (tags) to add to existing layers, or simply make a PR.
 
+## Generating docs
+
+If you add a new layer or modify an existing layer, run `schema.sh` to regenerate [SCHMEA.md](SCHEMA.md).
+
 ## Reporting problems
 
 If you find an issue with Layercake (such as certain features being processed incorretly), please open a bug on GitHub. If there's an error _in the data itself_, that is most likely a problem with the upstream OpenStreetMap data. You can check the data yourself on [openstreetmap.org](https://openstreetmap.org), and edit the map to fix the issue if it hasn't already been fixed.

README.md

@@ -4,7 +4,7 @@ Layercake is a set of thematic extracts of [OpenStreetMap](https://www.openstree
 
 OpenStreetMap’s native file format is OSM PBF, but this 80GB ‘planet file’ is unwieldy and not supported by all GIS software. Layercake provides OSM data separated into thematic layers (buildings, transportation, etc) and converted to cloud-native file formats like GeoParquet that are easy to use with software from DuckDB to QGIS.
 
-This repository contains the code that is used to generate the extracts, which are hosted by [OpenStreetMap US](https://openstreetmap.us/) at `data.openstreetmap.us`. [Instructions on how to access the data](https://openstreetmap.us/our-work/layercake/) are available on the OpenStreetMap US website.
+This repository contains the code that is used to generate the extracts, which are hosted by [OpenStreetMap US](https://openstreetmap.us/) at `data.openstreetmap.us`. [Instructions on how to access the data](https://openstreetmap.us/our-work/layercake/) are available on the OpenStreetMap US website. The schema produced by layercake is availabe in [SCHEMAS.md](SCHEMA.md).
 
 > [!WARNING]
 > Layercake is still experimental, and may change as it evolves. If you are interested in thematic extracts of OSM data, you can help the project's development by using it and providing feedback.

SCHEMA.md

@@ -0,0 +1,32 @@
+# Layercake schema
+
+Generated from the SQL layer definitions in [`sql/`](sql/).
+
+For an exact version of the deployed schema, use duckdb to check ex.
+
+`DESCRIBE SELECT * FROM 'https://data.openstreetmap.us/layercake/parks.parquet';`
+
+## addresses
+
+`type`, `id`, `addr:housenumber`, `addr:housename`, `addr:conscriptionnumber`, `addr:streetnumber`, `addr:provisionalnumber`, `addr:flats`, `addr:unit`, `addr:floor`, `addr:door`, `addr:street`, `addr:place`, `addr:city`, `addr:postcode`, `addr:hamlet`, `addr:district`, `addr:suburb`, `addr:neighbourhood`, `addr:quarter`, `addr:block_number`, `addr:block`, `addr:village`, `addr:town`, `addr:subdistrict`, `addr:county`, `addr:province`, `addr:state`, `addr:country`, `addr:full`, `bbox`, `geometry`
+## boundaries
+
+`type`, `id`, `boundary`, `admin_level`, `name`, `names`, `official_name`, `official_names`, `int_name`, `alt_name`, `alt_names`, `place`, `border_type`, `ISO3166-2`, `ISO3166-1:alpha2`, `ISO3166-1:alpha3`, `wikidata`, `wikipedia`, `disputed_by`, `claimed_by`, `controlled_by`, `recognized_by`, `bbox`, `geometry`
+## buildings
+
+`type`, `id`, `building`, `building:levels`, `building:flats`, `building:material`, `building:colour`, `building:part`, `building:use`, `name`, `website`, `wikipedia`, `wikidata`, `height`, `roof:shape`, `roof:levels`, `roof:colour`, `roof:material`, `roof:orientation`, `roof:height`, `start_date`, `access`, `wheelchair`, `bbox`, `geometry`
+## highways
+
+`type`, `id`, `highway`, `service`, `crossing`, `cycleway`, `cycleway:left`, `cycleway:right`, `footway`, `construction`, `name`, `ref`, `bridge`, `covered`, `lanes`, `layer`, `lit`, `sidewalk`, `smoothness`, `surface`, `tracktype`, `tunnel`, `wheelchair`, `width`, `access`, `bicycle`, `bus`, `foot`, `hgv`, `maxspeed`, `motor_vehicle`, `motorcycle`, `oneway`, `toll`, `bbox`, `geometry`
+## parks
+
+`type`, `id`, `boundary`, `protected_area`, `leisure`, `name`, `names`, `short_name`, `short_names`, `official_name`, `official_names`, `protect_class`, `protection_title`, `protected`, `iucn_level`, `access`, `operator`, `operator:type`, `owner`, `ownership`, `start_date`, `related_law`, `website`, `wikidata`, `wikipedia`, `bbox`, `geometry`
+## pois
+
+`type`, `id`, `amenity`, `attraction`, `club`, `craft`, `education`, `healthcare`, `historic`, `leisure`, `office`, `playground`, `shop`, `tourism`, `landuse`, `natural`, `name`, `names`, `official_name`, `official_names`, `old_name`, `old_names`, `alt_name`, `alt_names`, `short_name`, `short_names`, `brand`, `brand:wikidata`, `operator`, `operator:wikidata`, `phone`, `email`, `website`, `wikidata`, `wikipedia`, `access`, `bar`, `bicycle_parking`, `building`, `check_date`, `check_dates`, `cuisine`, `foods`, `drinks`, `diets`, `denomination`, `description`, `emergency`, `fountain`, `healthcare:speciality`, `heritage`, `nursery`, `opening_hours`, `preschool`, `recycling_type`, `religion`, `self_service`, `shelter`, `social_facility`, `social_facility:for`, `source`, `sport`, `wheelchair`, `bbox`, `geometry`
+## settlements
+
+`type`, `id`, `place`, `name`, `names`, `alt_name`, `alt_names`, `official_name`, `official_names`, `wikidata`, `wikipedia`, `population`, `bbox`, `geometry`
+## waterways
+
+`type`, `id`, `waterway`, `name`, `names`, `ref`, `intermittent`, `layer`, `bridge`, `tunnel`, `access`, `usage`, `seasonal`, `tidal`, `width`, `depth`, `oneway`, `lock`, `lock`, `lock`, `motorboat`, `ship`, `sailboat`, `boat`, `canoe`, `open_water`, `narrow`, `rapids`, `rapids:name`, `hazard`, `bbox`, `geometry`

schema.sh

@@ -0,0 +1,100 @@
+#!/bin/sh
+
+# Generate Markdown documentation for the columns produced by each SQL layer.
+# Usage: ./schema.sh
+
+set -eu
+
+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+SQL_DIR="$SCRIPT_DIR/sql"
+OUTPUT="$SCRIPT_DIR/SCHEMA.md"
+
+TMP_OUTPUT=$(mktemp "$OUTPUT.tmp.XXXXXX")
+cleanup() {
+    rm -f "$TMP_OUTPUT"
+}
+trap cleanup EXIT HUP INT TERM
+
+{
+    printf '%s\n' '# Layercake schema' '' \
+        'Generated from the SQL layer definitions in [`sql/`](sql/).' '' \
+        'For an exact version of the deployed schema, use duckdb to check ex.' '' \
+        '`DESCRIBE SELECT * FROM '\''https://data.openstreetmap.us/layercake/parks.parquet'\'';`' ''
+
+    found_layer=0
+    for sql_file in "$SQL_DIR"/*.sql; do
+        [ -f "$sql_file" ] || continue
+
+        table=$(basename "$sql_file" .sql)
+        [ "$table" = 'macros' ] && continue
+        found_layer=1
+
+        printf '## %s\n\n' "$table"
+
+        awk '
+            # Remove whitespace around an extracted column name.
+            function trim(value) {
+                sub(/^[[:space:]]+/, "", value)
+                sub(/[[:space:]]+$/, "", value)
+                return value
+            }
+
+            # The second SELECT is the layer output schema. Start reading
+            # columns there rather than the SELECT in the raw CTE.
+            /^[[:space:]]*SELECT[[:space:]]*$/ {
+                in_output_select = 1
+                next
+            }
+
+            # The output column list ends immediately before FROM raw.
+            in_output_select && /^[[:space:]]*FROM[[:space:]]+raw[[:space:]]*$/ {
+                exit
+            }
+
+            in_output_select {
+                # Normalize the line before examining the selected expression.
+                line = $0
+                sub(/^[[:space:]]+/, "", line)
+                sub(/[[:space:]]*,[[:space:]]*$/, "", line)
+
+                # Ignore comments and the fields inside the bbox struct.
+                if (line ~ /^--/ || line ~ /^[{]/ || line ~ /^}[[:space:]]*$/ || line == "")
+                    next
+
+                # type, id, and geometry are selected without aliases.
+                if (line == "type" || line == "id" || line == "geometry") {
+                    column = line
+                # For expressions, keep the name following AS as the column.
+                } else if (line ~ /[[:space:]]+AS[[:space:]]+/) {
+                    sub(/^.*[[:space:]]+AS[[:space:]]+/, "", line)
+                    column = trim(line)
+                    # SQL permits aliases surrounded by single or double
+                    # quotes (and backticks), which are not part of the name.
+                    quote = sprintf("%c", 39)
+                    if (substr(column, 1, 1) == quote || substr(column, 1, 1) == "\"" || substr(column, 1, 1) == "`")
+                        column = substr(column, 2)
+                    if (substr(column, length(column), 1) == quote || substr(column, length(column), 1) == "\"" || substr(column, length(column), 1) == "`")
+                        column = substr(column, 1, length(column) - 1)
+                } else {
+                    next
+                }
+
+                if (column_count++ > 0)
+                    printf ", "
+                printf "`%s`", column
+            }
+
+            END {
+                printf "\n"
+            }
+        ' "$sql_file"
+    done
+
+    if [ "$found_layer" -eq 0 ]; then
+        printf '%s\n' '_No SQL layer files found._'
+    fi
+} > "$TMP_OUTPUT"
+
+mv "$TMP_OUTPUT" "$OUTPUT"
+trap - EXIT HUP INT TERM
+printf 'Generated %s\n' "$OUTPUT"