Skip to content

[Draft] Atlanta scene-combination benchmark: run_benchmark.sh + study notebook (#169) - #198

Draft
bpurinton wants to merge 1 commit into
issue-169-dem-benchmarkfrom
issue-169-benchmark-runs
Draft

[Draft] Atlanta scene-combination benchmark: run_benchmark.sh + study notebook (#169)#198
bpurinton wants to merge 1 commit into
issue-169-dem-benchmarkfrom
issue-169-benchmark-runs

Conversation

@bpurinton

Copy link
Copy Markdown
Contributor

Draft, stacked on #197 (base issue-169-dem-benchmark; GitHub will retarget it to main when #197 merges). The processing script and the study notebook for the systematic part of #169no stereo runs have been launched; that is the one decision this PR is waiting on.

What's in it

  • notebooks/WorldView/run_benchmark.sh — idempotent (every step skipped when its product exists), takes the data directory as its argument, identical stereo settings to run_mvs.sh / run_pairwise.sh, ASP bin appended to PATH. Produces, in order:

    1. wv_correct × 5 (the *_corr.tif intermediates were purged; two were dangling symlinks — handled).
    2. The seven remaining pairs → all ten pairs, convergence 5–32°.
    3. dem_mosaic of all ten pairs and of the six pairs > 15° (13-8, 8-16, 13-10, 8-21, 10-16, 10-21) — the test of whether the mosaic's problem is averaging, or averaging weak pairs in.
    4. A nested quad {13,10,16,21} (pair 13-10 ⊂ MVS3 ⊂ quad ⊂ MVS5) and a wide-spread triple {13,8,21}.

    ≈ 11–12 h on the laptop (pair ≈ 1 h, triple ≈ 2 h, quad ≈ 2.5 h), ~30 GB. The new left-image crop windows (nadir8 5461 9789 13354 12980, nadir16 6083 8756 12879 11461, nadir21 6467 8177 12495 10687) were derived from the BA clean matches inside the nadir13 crop exactly as CROP10 was — and the method reproduces CROP10 (5540 13223 13206 12931) bit for bit.

  • notebooks/WorldView/worldview_spacenet_atlanta_benchmark.ipynb — the study, built so it is valid at every stage: it parses ba/run-convergence_angles.txt into the pair table, reproduces the crop-window derivation, documents the script, then scores whatever DEMs exist on disk with DEMBenchmark (listing the rest as pending) and draws the three analyses — NMAD vs. convergence angle over the pairs, the 2→3→4→5 nested-scene curve, and the MVS-vs-mosaic flow table at 3 and 5 scenes. Executed as committed against the six existing DEMs, so the figures are partial (3 of 10 pairs, 3 of 4 chain points) by design.

Not yet

  • No docs card / toctree entry and no CHANGELOG line — those come when the runs exist and the notebook is complete.
  • multi_stereo --mode dem_mosaic (ASP's own pairwise+mosaic flow, unreleased) is not in the local 3.8.0-alpha build; if a newer daily build is available it would be a good extra row.

To launch

bash notebooks/WorldView/run_benchmark.sh ~/Desktop/asp-plot-examples/atlanta_mvs 2>&1 | tee ~/Desktop/asp-plot-examples/atlanta_mvs/processing_benchmark.log

Then re-execute the notebook; the pending lists empty themselves and the write-up gets finished.

The systematic half of #169: given five same-pass WorldView-2 scenes,
which combinations and which processing flow give the best DEM. The
six DEMs scored in the MVS notebook answer it for one triple; the full
matrix needs the seven remaining pairs (all ten span 5-32 degrees of
convergence), a nested quad for the 2->3->4->5 scene curve, a
wide-spread triple, and dem_mosaic of all ten pairs and of the six
well-converged ones.

run_benchmark.sh produces exactly those, with the settings of the
existing runs and every step skipped when its product exists. The
left-image crop windows for nadir8/16/21 were derived from the
bundle_adjust clean matches inside the nadir13 crop the way
run_pairwise.sh derived nadir10's; the derivation reproduces nadir10's
window exactly, which is the check on the method. The two dangling
*_corr.tif symlinks left by the earlier cleanup are handled.

The notebook is built to be valid at every stage: it scores whatever
DEMs exist on disk with DEMBenchmark and lists the rest as pending, so
as committed it covers the six existing DEMs (3 of 10 pairs, 3 of 4
chain points) and fills itself in once the runs land. No runs have been
launched; that is the decision this branch waits on. No docs card or
CHANGELOG line until the study is complete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant