DEMBenchmark: score many DEMs against one altimetry sample (#169) - #197
Open
bpurinton wants to merge 3 commits into
Open
DEMBenchmark: score many DEMs against one altimetry sample (#169)#197bpurinton wants to merge 3 commits into
bpurinton wants to merge 3 commits into
Conversation
The report scores one DEM against ICESat-2 or LOLA/MOLA. Issue #169 asks the question that needs many: which scene combinations and which processing flow (joint multi-view triangulation vs. pairwise stereo + dem_mosaic) give the best DEM. The Atlanta MVS notebook answered it for two DEMs with a hand-rolled loop over Altimetry objects and an ad-hoc pandas table; this productizes that loop so the next comparison is one call instead of a page of notebook code. asp_plot/dem_benchmark.py: DEMBenchmark takes {label: dem_fn} plus one parquet cache (or planetary CSV) and scores every DEM with the report's recipe -- replayed points, WorldCover water filter, 3-sigma cut -- into a one-row-per-DEM stats table: coverage inside the common footprint (the intersection of all DEM extents, so different crop windows compare fairly), IntersectionErr median/NMAD from the point2dem sibling raster, altimetry-minus-DEM median/NMAD/RMSE before and after a per-DEM pc_align --compute-translation-only, and an optional difference against one candidate named as the reference. summary_plot() draws the table as dumbbell panels sorted best-first; histogram_plot() overlays the residual distributions. A dem_benchmark CLI wraps it (LABEL=PATH specs, --parquet / --altimetry-csv, --reference, --no-pc-align). pc_align products and the translated DEM copies go under <directory>/dem_benchmark/<label>/ rather than next to each candidate, which needed one backward-compatible addition: an output_fn argument on Alignment.apply_dem_translation() (default unchanged). Existing pc_align outputs are reused so a re-run is instant and offline; a missing pc_align binary degrades to pre-alignment scoring with a warning. Verified on the six Atlanta DEMs (3 pairs, MVS3, MVS5, 3-pair mosaic): 74 s end to end, and the numbers reproduce the notebook (MVS3 NMAD 0.71 vs mosaic 1.02 m). Tests run offline on the Utqiagvik fixtures with the pc_align subprocess stubbed by the real log fixture.
The Atlanta MVS notebook compared its two DEMs (3-scene MVS vs. the 3-pair dem_mosaic) with a hand-rolled loop over Altimetry objects and an ad-hoc pandas table. It now uses DEMBenchmark, and scores six DEMs on the same ICESat-2 sample: the two workflows, each of the three single pairs behind the mosaic, and the 5-scene MVS run that #159 had reserved for this work. That answers two things the two-DEM comparison could only guess at. The single pairs explain the mosaic: the 26.9 and 21.8 degree pairs score NMAD 0.64 and 0.72 m, the 5.1 degree pair 1.54 m, and dem_mosaic averages all three to 1.02 m -- below either good pair alone. And the 5-scene run wins on bias (+0.12 m) rather than spread (0.68 m); its five-ray IntersectionErr median (0.19 m) reads higher than any pair's, which is why the notebook now says that a narrow pair's small triangulation error is not a quality ranking. After pc_align every median lands at +0.16..0.27 m and no NMAD moves. The systematic matrix stays with #169. Also: the dem_benchmark CLI page (docs/cli/dem_benchmark.md) with the notebook's figure as its example, the CLI index (seven tools), ARCHITECTURE.md entries for the module and the command, the AGENTS.md CLI list, and the 3.3.0 CHANGELOG entry. The notebook was re-executed with the report cell skipped (the report is unchanged by this work); its outputs are preserved. pc_align products for the six candidates live under atlanta_mvs/dem_benchmark/.
A local Sphinx build flagged the only three warnings in the package as coming from the new module: "Inline emphasis start-string without end-string" from the bare *-IntersectionErr.tif in the Click help (RST reads the asterisk as emphasis), and two "duplicate object description" warnings because the class docstring's numpydoc Attributes section and the instance attributes assigned in __init__ both produced entries for stats_df and altimetry. The help text now says "IntersectionErr raster" (the docs page's --help dump is refreshed to match), and the Attributes section is prose under Notes, which documents the same three attributes without emitting directives that collide with autoapi's own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First piece of #169: a way to score many DEMs against one altimetry sample, so scene combinations and processing flows (joint MVS vs. pairwise +
dem_mosaic) can be compared apples to apples. The report assesses one DEM; the Atlanta MVS notebook had been comparing two with a hand-rolled loop. This makes that loop a class, a figure, and a CLI, and uses them in the notebook to score six DEMs at once.No new dependencies. One new entry point (
dem_benchmark) — the conda-forge recipe in this PR is updated, and the feedstock will need it synced by hand at release (the autotick bot doesn't).What's in it
asp_plot/dem_benchmark.py—DEMBenchmark.{label: dem_fn}+ the report's parquet cache (or a LOLA/MOLA CSV) → one row per DEM (stats_df): coverage inside the common footprint (intersection of all DEM extents, so different crop windows compare fairly), IntersectionErr median/NMAD from thepoint2demsibling (NaN for a mosaic — itself a finding), altimetry−DEM n/median/NMAD/RMSE before and after a per-DEMpc_align --compute-translation-only, and an optional difference against one candidate named as reference. Same recipe as the report: replayed points, WorldCover water filter, 3σ cut.summary_plot()(rows sorted best-first, before→after dumbbells) andhistogram_plot(); oneAltimetryper DEM is kept for follow-up per-DEM figures.<directory>/dem_benchmark/<label>/, reused on re-run (instant, offline). Needed one backward-compatible addition:Alignment.apply_dem_translation(output_fn=...). A missingpc_alignbinary degrades to pre-alignment scoring with a warning.dem_benchmarkCLI (LABEL=PATHspecs,--parquet/--altimetry-csv,--reference,--no-pc-align,--own-extent) → summary PNG +_histogram.png+ stats CSV. Docs pagedocs/cli/dem_benchmark.md, CLI index (seven tools), ARCHITECTURE/AGENTS/CHANGELOG (3.3.0) entries.tests/test_dem_benchmark.py, 19, offline): Utqiagvik fixtures,pc_alignsubprocess stubbed with the real log fixture; labels, validation, coverage/IE/dh columns, alignment reuse, minimum-points and missing-binary paths, non-overlapping DEM, figures, CLI smoke.What the six Atlanta DEMs say
dem_mosaicaverages a 1.54 m pair with two ~0.7 m pairs and lands at 1.02 m — below either good pair alone. MVS's win over the mosaic is real (≈30% NMAD), but its edge over the best single pair is bias, not spread.Scope / not in this PR
multi_stereo --mode dem_mosaiconce a build has it) is the follow-up on Benchmark scene combinations and MVS vs. n-pair + merge flows against altimetry #169 and will use this tool as-is.asp_reportPDF (likegallery).Altimetry+ parquet +atl06sr_to_dem_dhprints "Error in sys.excepthook" lines at interpreter exit (exit code 0). It shows up once per DEM here because there are N of them; reproduced with the bare package API and left alone.Verified
dem_benchmarkon the six Atlanta DEMs: 74 s end to end; numbers reproduce the notebook's previous two-DEM comparison (MVS3 0.71 vs mosaic 1.02 m NMAD; DEM-to-DEM +0.02 m / 0.51 m).Closes nothing yet; #169 stays open for the benchmark runs.