diff --git a/.gitmodules b/.gitmodules index 4981a746f..48d6b3e03 100644 --- a/.gitmodules +++ b/.gitmodules @@ -13,3 +13,6 @@ [submodule "thirdparty/FastVGGT"] path = thirdparty/FastVGGT url = https://github.com/mystorm16/FastVGGT.git +[submodule "thirdparty/vggt-omega"] + path = thirdparty/vggt-omega + url = https://github.com/facebookresearch/vggt-omega.git diff --git a/REVIEW.md b/REVIEW.md new file mode 100644 index 000000000..20f78048d --- /dev/null +++ b/REVIEW.md @@ -0,0 +1,151 @@ +# PR #1117 — VGGT Verified Pipeline + +**16 commits · +1913 / −80 · 23 files · targets `mapping-vggt`** + +This branch turns the per-cluster VGGT reconstruction into a **verified-view-graph pipeline** and stacks +four accuracy/robustness features on top of it, ending with a **VGGT-Omega** geometry arm. It's large +because it was developed as one experimental thread; this doc decomposes it into **6 self-contained +feature units** that map 1:1 to the clean branches we'll land into `master`. + +## Headline result — IMC Phototourism *Grand Place Brussels* (234 images) + +| Arm (config) | AUC@3° (full) | AUC@3° (constructed) | Cameras | +|---|---|---|---| +| Prior VGGT baseline (non-verified) | ~0.675 | — | ~219 | +| **VGGT — verified** (`vggt_sift_…_verified`) | **0.6968** | 0.7213 | **230** | +| **VGGT-Omega — verified** (`vggt_omega_…_verified`) | **0.7191** | **0.7380** | **231** | + +Controlled A/B: the omega vs VGGT rows differ **only** in the per-cluster geometry model — same frontend, +partition, Fetzer, and retri. + +## How to review this efficiently + +1. **Start with the configs** — they are the table of contents. Diff + `gtsfm/configs/vggt_sift_frontend_megaloc_phototourism_verified.yaml` (every new flag is commented in place). +2. **Then `scene_optimizer.py`** (+205) — the orchestration: global two-view verification → verified + partition → global Fetzer → post-merge retriangulation. +3. **Then the per-feature files** in the order below. +4. **Mechanical vs load-bearing:** `cluster_vggt_omega_with_frontend.py` (+404) and + `vggt_omega_geometry_transformer.py` (+347) are **vendored from PR #1116** (Harneet) — review at the + interface level, not line-by-line. The load-bearing *new* logic is in `scene_optimizer.py`, + `cluster_merging.py`, `cluster_vggt_with_frontend.py`, and `view_graph_calibration.py`. + +## New config flags (the feature switches — all default to the prior behavior) + +| Flag | Where | Default | Effect | +|---|---|---|---| +| `use_verified_pipeline` | top-level | `false` | global two-view verification + verified-graph METIS partition + post-merge retriangulation | +| `use_view_graph_calibration` | optimizer | `false` | per-edge scipy Fetzer focal refinement | +| `use_global_view_graph_calibration` | optimizer | `false` | single **global** Fetzer over the verified graph (takes precedence) | +| `calibration_prior_focal_sigma` | ba_options | n/a | anchors focals in cluster (5px) & merge (10px) BAs | +| `reuse_global_correspondences` | optimizer | `false` | build per-cluster tracks from the global verified frontend (skip per-cluster frontend) — pure speedup | +| `recover_trackless_cameras_in_retriangulation` | merging | `false` | inject good-pose/no-track cams into the post-merge retri for a geometric second chance | + +**Every flag off ⇒ byte-identical to the prior baseline.** This is the key reviewer reassurance: the diff +is additive and gated. + +--- + +## Feature units → suggested clean branches (with landing order) + +**Dependency order: ① must land first** (everything reads its `ClusterContext` fields + `use_verified_pipeline`). +**②③④⑤ are independent** of each other on top of ①. **⑥ is independent** but its config overlays ①–⑤. + +### ① Verified view-graph pipeline *(foundation)* +- **Commits:** `aeba561a`, `86a5a3ca`, `1c5ef76f`, `1dc132ba` +- **What:** Runs one global two-view verification pass over the full MegaLoc retrieval graph, partitions + METIS on the **verified** subgraph (edges where `TwoViewResult.valid()`), and adds a **post-merge + retriangulation+BA** stage (`results/merged_retriangulated/`). Several follow-ups fix worker-OOM/dask-nesting + by running the global frontend **inline** instead of via nested `worker_client()` submission. +- **Key files:** `scene_optimizer.py` (orchestration), `cluster_mvo.py`, `two_view_estimator.py` + (`create_two_view_results_inline`), `frontend/correspondence_generator/*` (`generate_correspondences_inline`), + `cluster_optimizer_base.py` (new `ClusterContext` fields). +- **Review focus:** the inline-vs-dask execution model (the OOM fixes) and the verified-graph construction. + +### ② Global Fetzer focal calibration + focal-flow anchoring +- **Commits:** `e08be7c8`, `20adfc40`, `7b641621` +- **What:** Estimates focals with scipy Fetzer over the verified F-matrices (`compute_global_view_graph_intrinsics`), + then **anchors** those focals through the cluster BA (σ=5px) and merge BA (σ=10px) so BAs optimize + *poses, not focals* (kills the focal/depth ambiguity that was diverging separator cameras across clusters). +- **Key files:** `graph_optimizer/view_graph_estimator/view_graph_calibration.py` (+44), `scene_optimizer.py`, + the `ba_options.calibration_prior_*` plumbing. +- **Note:** Fetzer initializes from the loader heuristic (1.2·maxdim) and **never reads VGGT/omega focals** — + important for the omega arm (its anisotropic fx/fy is irrelevant; a single-focal `Cal3Bundler` is produced). + +### ③ Peak frontend (config + PoseLib verifier) +- **Commits:** `170dc610`, `77796d20` +- **What:** Adopts the "gp-glomap-parity" frontend — `PoseLibVerifier` (5-point + LO-RANSAC), 8192 ColmapSIFT + keypoints, 30/0.15 inlier gate — and retunes the Brussels baseline + verified configs. +- **Key files:** `configs/vggt_sift_frontend_megaloc_phototourism.yaml` (+ verified), `two_view_estimator.py`, + **`pyproject.toml`** (+`poselib>=2.0`). +- **Review focus:** the new `poselib` dependency + verifier swap. Mostly config; lowest-risk unit. + +### ④ Global correspondence reuse *(speedup)* +- **Commits:** `1bc73192`, `ede5a9ba`, `8d6948ba` +- **What:** Each cluster builds its 2D tracks from the **already-computed** global verified correspondences + instead of re-running its own frontend (which only cache-read the same data, serially + redundantly across + overlapping clusters). Pure speedup — per-edge `v_corr` is identical. *(One commit, `ede5a9ba`, reverts an + experimental per-cluster triangulated-structure path that inflated clusters to 11k–15k tracks → OOM; it keeps + the focal-flow fix. See "reverted experiments" below.)* +- **Key files:** `cluster_vggt_with_frontend.py` (the `reuse_global_correspondences` branch), + `ClusterContext.global_v_corr_idxs_dict` / `global_keypoints`. +- **Cache token:** adds `/gcorr` to the optimizer `__repr__` (cluster-cache key). + +### ⑤ Trackless-camera recovery +- **Commits:** `bbd79ff7` (build-side "all-measurements"/`/allkpts`), `1af1e43e` (retriangulation recovery) +- **What:** Two orthogonal levers to rescue cameras that get good poses but no VGGT-depth track. (a) The + depth-lift build no longer lets zero-confidence VGGT depth *veto* a verified SIFT measurement (`/allkpts`). + (b) Cameras dropped at the root merge's track filter are **captured and injected** into the post-merge + retriangulation for a geometric (depth-independent) second chance; those that still can't triangulate are + cleanly dropped. **Provably can't perturb the constructed cameras** (recovered cams have <15 tracks ⇒ + excluded from the retri BA factor graph). +- **Key files:** `cluster_merging.py` (+38, capture + `trackless_cameras` on the merge result), + `scene_optimizer.py` (`_run_post_merge_retriangulation` inject + first-class diagnostic logging), + `cluster_vggt_with_frontend.py` (build). +- **Result note:** on Brussels this recovers cam 49; 104/207 remain (they're a pose-constraint problem — see + follow-ups). + +### ⑥ VGGT-Omega geometry integration +- **Commits:** `dc9824af` (integration), `7a7c51ca` (preprocessing-dispatch fix) +- **What:** Vendors VGGT-Omega (from PR #1116) and runs it **through our verified optimizer** + (`ClusterVGGTWithFrontend` via its injected `geometry_transformer`), *not* the bundled + `ClusterVGGTOmegaWithFrontend` (a master-based copy lacking Fetzer/reuse/recovery). Two small enabling changes: + (a) guard `transformer.config` access (omega has none) at `__init__` + `__repr__`; (b) **generalize image + preprocessing to follow the transformer** — new `GeometryTransformer.load_image_batch` (base = VGGT loader, + omega overrides → its own 512/16-aligned loader). This is the cleanest standalone refactor in the PR and is a + **no-op for all VGGT configs**. +- **Key files:** `frontend/geometry_transformer.py` (+20, the abstraction), + `frontend/vggt_omega_geometry_transformer.py` (+347, vendored), + `cluster_optimizer/cluster_vggt_omega_with_frontend.py` (+404, vendored), `cluster_vggt.py` (loader dispatch), + `cluster_optimizer/__init__.py` (register), `configs/vggt_omega_*` (×2), `.gitmodules` + + `thirdparty/vggt-omega` submodule, `scripts/download_model_weights.sh`, `utils/torch.py`. +- **⚠️ Reviewer/ops notes:** weights are **gated, `cc-by-nc-4.0`, non-redistributable**, **CUDA-only**. + Submodule + `--hf_token` download required; omega module imports are lazy so non-omega paths/tests are + unaffected. **The geometry-transformer abstraction (a) is worth landing as its own tiny branch first** — + omega then becomes a pure add-on. + +--- + +## Reverted experiments (intentionally *not* in the final pipeline) +- **Per-cluster triangulated structure** (`use_triangulated_structure: true`) — inflated clusters to 11k–15k + tracks → 85–200s BAs → worker OOM. Reverted to VGGT depth-lift (`ede5a9ba`); the flag remains but defaults + `false`. + +## Known limitations / planned follow-ups (out of scope for this PR) +1. **Brussels 104/207 still unrecovered** — they never acquire tracks at *any* stage, so they're carried + through the Sim3 merge unconstrained (one is ~90°-flipped). This is a **pose-constraint problem → + PnP/resection**, not a geometry-model one (omega didn't and couldn't fix it). +2. **One Sim3-flipped camera** (max rot err 90.05° vs 0.13° median) dominates the mean rotation error — + separate AUC lever. +3. **Omega licensing/ops** — NC weights must stay out of any commercial/redistributed artifact. + +## How to run / verify +```bash +git submodule update --init --recursive # omega arm only +bash scripts/download_model_weights.sh --hf_token # omega arm only (CUDA) +uv run ./run --config_name vggt_omega_sift_frontend_megaloc_phototourism_verified \ + loader._target_=gtsfm.loader.Colmap loader.dataset_dir=$DATA \ + +loader.colmap_files_subdir=sfm_updated +loader.use_gt_intrinsics=false +``` +Drop `_omega` from the config name for the VGGT arm. Metrics land in `results/merged_retriangulated/…json` +(`pose_auc_@3.0_deg`, `number_cameras_merged`). diff --git a/gtsfm/bundle/bundle_adjustment.py b/gtsfm/bundle/bundle_adjustment.py index b89fe0a0b..37f91e05d 100644 --- a/gtsfm/bundle/bundle_adjustment.py +++ b/gtsfm/bundle/bundle_adjustment.py @@ -158,6 +158,10 @@ class BundleAdjustmentOptions: use_calibration_prior: bool = False use_pose_prior_all_cameras: bool = False use_pose_prior_first_camera: bool = False + # Sigma for the PriorFactorPose3 added by the use_pose_prior_* flags, anchored at the + # initialization poses. Near-zero (e.g. 1e-3, in scene units) freezes poses so the BA + # refines only structure + calibration ("self-calibration against the init poses"). + cam_pose3_prior_noise_sigma: float = 0.1 use_gnc: bool = False gnc_loss: Union[RobustBAMode, str] = RobustBAMode.GMC factor_weight_outlier_threshold: float = 0.0 @@ -188,6 +192,7 @@ def to_optimizer(self, **overrides) -> "BundleAdjustmentOptimizer": use_calibration_prior=self.use_calibration_prior, use_pose_prior_all_cameras=self.use_pose_prior_all_cameras, use_pose_prior_first_camera=self.use_pose_prior_first_camera, + cam_pose3_prior_noise_sigma=self.cam_pose3_prior_noise_sigma, use_gnc=self.use_gnc, gnc_loss=self.gnc_loss, factor_weight_outlier_threshold=self.factor_weight_outlier_threshold, diff --git a/gtsfm/cluster_merging.py b/gtsfm/cluster_merging.py index b10008d7a..506fc7d1f 100644 --- a/gtsfm/cluster_merging.py +++ b/gtsfm/cluster_merging.py @@ -17,7 +17,6 @@ import gtsfm.utils.logger as logger_utils import gtsfm.utils.metrics as metrics_utils from gtsfm.bundle.bundle_adjustment import BundleAdjustmentOptions -from gtsfm.cluster_optimizer.cluster_anysplat import save_splats from gtsfm.common.gtsfm_data import GtsfmData from gtsfm.evaluation.metrics import GtsfmMetric, GtsfmMetricsGroup from gtsfm.utils import align as align_utils @@ -36,6 +35,12 @@ _SCENE_PLOTS_DIR_ATTR = "_gtsfm_plots_dir" _SCENE_LABEL_ATTR = "_gtsfm_cluster_label" +# Sidecar attached to each scene: packed (camera, pixel) -> global-track-id index (see +# build_measurement_gid_arrays). Rides the GtsfmData through dask exactly like the plots/label attrs, +# so merges can recognize that two locally-triangulated tracks are the SAME global track even when the +# two reconstructions share no cameras (the identity was established once, globally, by the verified +# union-find; per-cluster slicing used to discard it). +_SCENE_GID_INDEX_ATTR = "_gtsfm_measurement_gid_index" @dataclass @@ -48,6 +53,14 @@ class MergingOptions: drop_outlier_after_camera_merging: bool = True drop_camera_with_no_track: bool = True keep_all_cameras: bool = False + # MERGE_GUARD Sim3 scale band — DISABLED by default (band = [0, inf)). VGGT scene scale is + # arbitrary PER CLUSTER (per-batch normalization), so heterogeneous cluster extents produce + # large-but-lawful solved scales: the old [0.25, 4] band executed 18 lawful Roman Forum children + # (~1,200 cams incl. a 258-cam subtree at scale 4.06). Genuinely diverged seats are caught by + # the structureless/0-track guards, the correspondence floor, and the post-merge BA + reproj + # filters. Set a finite band only to re-enable the legacy behavior. + sim3_scale_band_min: float = 0.0 + sim3_scale_band_max: float = float("inf") plot_reprojection_histograms: bool = True use_nonlinear_sim3_alignment: bool = False max_track_correspondences_for_sim3: int = 150 @@ -58,7 +71,31 @@ class MergingOptions: min_track_length: int = 2 allow_post_ba_reproj_filtering: bool = True metric_constructed_only: bool = False + # When set, the root merge captures the good-pose cameras that the post-BA track filter drops + # (e.g. low VGGT-depth-confidence cams left track-less at construction) and hands their merged + # poses to the post-merge retriangulation, which geometrically re-triangulates their global 2D + # tracks against those poses. Cameras that still fail to gain a >=3-view track are cleanly dropped + # by the retri's own final filter. Orthogonal to the per-cluster depth gate: an unrecovered camera + # is inert (excluded from the retri BA, contributes zero factors), while a recovered good-pose camera + # (>=15 inlier tracks) joins the retri BA and jointly refines shared structure as intended — all poses + # pinned by use_pose_prior_all_cameras, so the existing reconstruction is bounded, not free to drift. + recover_trackless_cameras_in_retriangulation: bool = False + # Global-track-ID merge guard (active only when scenes carry the gid-index sidecar, i.e. the verified + # pipeline): a child with >= guard_child_min_cams cameras whose ID-matched Sim3 correspondences come out + # below min_sim3_correspondences_large_child has no reliable structural tie to the parent — its 7-DoF + # seat would be under-constrained (ToL: an 83-cam child seated on 26 correspondences produced a rigid + # 68-camera stray block 60-190m off). Such children are DROPPED (accuracy over camera count). + guard_child_min_cams: int = 30 + min_sim3_correspondences_large_child: int = 50 ba_options: BundleAdjustmentOptions = field(default_factory=BundleAdjustmentOptions) + # Post-merge retriangulation BA mode. When retri_free_ba is True the retri BA drops the + # calibration/pose priors — a fully-free final solve (GLOMAP-style): the priors that protect the + # incremental merges throttle the full-scene BA, which otherwise locks in the accuracy (BM offline + # replay: constructed AUC@10 0.770->0.806 on the same input; robust kernel/GNC inherited from + # ba_options). retri_iterations>1 re-triangulates against the improved poses between BAs — only + # worth it when the merged input is poor. Defaults preserve existing behavior. + retri_free_ba: bool = False + retri_iterations: int = 1 def _create_unary_measurements(scene: GtsfmData) -> list[UnaryMeasurementPose3]: @@ -85,11 +122,206 @@ def _measurement_key(cam_idx: int, uv: np.ndarray) -> tuple[int, int, int]: return cam_idx, math.floor(float(uv[0])), math.floor(float(uv[1])) +# --------------------------------------------------------------------------- +# Global-track-ID (gid) index: packed numpy, NOT python dicts — lean enough to ride each scene +# (KB-MB per cluster, ~20MB at the root union) with zero extra gather/broadcast. Keys use the SAME +# integer-floor quantization as _measurement_key, and measurements survive BA/filtering verbatim, so a +# track's global identity is recoverable at ANY pipeline stage from any one of its measurements. +# --------------------------------------------------------------------------- + +_GID_CAM_SHIFT = 44 +_GID_U_SHIFT = 22 +_GID_COORD_MASK = (1 << 22) - 1 + + +def _pack_measurement_keys(cams: np.ndarray, us: np.ndarray, vs: np.ndarray) -> np.ndarray: + """Pack (camera, floor(u), floor(v)) into a single int64 (same quantization as _measurement_key).""" + u = np.clip(np.floor(us).astype(np.int64), 0, _GID_COORD_MASK) + v = np.clip(np.floor(vs).astype(np.int64), 0, _GID_COORD_MASK) + return (cams.astype(np.int64) << _GID_CAM_SHIFT) | (u << _GID_U_SHIFT) | v + + +def build_measurement_gid_arrays(tracks_2d) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + """Flatten global 2D tracks into parallel (packed_key, gid, cam) arrays (unsorted). + + Built ONCE in the main process from the global union-find tracks; per-cluster slices are cut with + slice_gid_index. ~1.8M entries / ~20MB for 337k Tower-of-London tracks. + """ + cams, us, vs, gids = [], [], [], [] + for gid, track in enumerate(tracks_2d): + for m in track.measurements: + cams.append(m.i) + us.append(float(m.uv[0])) + vs.append(float(m.uv[1])) + gids.append(gid) + cams_arr = np.asarray(cams, dtype=np.int64) + keys = _pack_measurement_keys(cams_arr, np.asarray(us), np.asarray(vs)) + return keys, np.asarray(gids, dtype=np.int32), cams_arr.astype(np.int32) + + +def slice_gid_index( + keys: np.ndarray, gids: np.ndarray, cams: np.ndarray, camera_set +) -> Optional[tuple[np.ndarray, np.ndarray]]: + """Cut the per-cluster gid index: entries whose camera is in ``camera_set``, sorted by key for lookup.""" + if keys is None or len(keys) == 0: + return None + mask = np.isin(cams, np.fromiter(camera_set, dtype=np.int32)) + if not mask.any(): + return None + k, g = keys[mask], gids[mask] + order = np.argsort(k, kind="stable") + return k[order], g[order] + + +def _lookup_gid(index: tuple[np.ndarray, np.ndarray], cam_idx: int, uv: np.ndarray) -> int: + """Return the global track id for one measurement, or -1 if unknown.""" + keys, gids = index + key = int( + _pack_measurement_keys(np.array([cam_idx]), np.array([float(uv[0])]), np.array([float(uv[1])]))[0] + ) + pos = int(np.searchsorted(keys, key)) + if pos < len(keys) and int(keys[pos]) == key: + return int(gids[pos]) + return -1 + + +def _track_gid(track: gtsam.SfmTrack, index: Optional[tuple[np.ndarray, np.ndarray]]) -> int: + """Global id of a 3D track = MAJORITY VOTE over ALL of its measurements found in the index (-1 if none). + + First-hit identity was dishonest: a single pixel-bin collision on the first indexed measurement + assigned the whole track a foreign gid, minting phantom parent<->child "correspondences" between + tracks that share no physical point (offline replay: zero-overlap children showed up with 150 + matches). Voting over every measurement makes rare collisions harmless — the true identity of the + remaining measurements outvotes them — so anchor counts (and the MERGE_GUARD decisions built on + them) are honest. + """ + if index is None: + return -1 + votes: dict[int, int] = {} + for m_idx in range(track.numberMeasurements()): + cam_idx, uv = track.measurement(m_idx) + gid = _lookup_gid(index, cam_idx, uv) + if gid >= 0: + votes[gid] = votes.get(gid, 0) + 1 + if not votes: + return -1 + return max(votes, key=votes.get) # type: ignore[arg-type] + + +def _scene_gid_index(scene: Optional[GtsfmData]) -> Optional[tuple[np.ndarray, np.ndarray]]: + return getattr(scene, _SCENE_GID_INDEX_ATTR, None) if scene is not None else None + + +def _union_gid_indices(*indices) -> Optional[tuple[np.ndarray, np.ndarray]]: + """Union of packed indices (first occurrence of a key wins); propagated up the merge tree.""" + live = [ix for ix in indices if ix is not None and len(ix[0]) > 0] + if not live: + return None + keys = np.concatenate([ix[0] for ix in live]) + gids = np.concatenate([ix[1] for ix in live]) + uniq_keys, first = np.unique(keys, return_index=True) + return uniq_keys, gids[first] + + +def _prefilter_point_pairs( + pairs: list[tuple[np.ndarray, np.ndarray]], spread: float +) -> tuple[list[tuple[np.ndarray, np.ndarray]], float, float]: + """RANSAC-verify point pairs BEFORE the joint Sim3 solve; returns (inliers, pair_scale, inlier_ratio). + + A wrong pair under a tight noise sigma dominates the quadratic cost and can warp the solve; worse, a + COHERENTLY wrong pair set (e.g. mismatched regions) implies a consistent wrong similarity that the + solver will follow off a cliff (observed: children with 150 correspondences solving at scale 81-289). + Fitting the pairs alone first both cleans them and measures the transform they actually imply, so the + caller can refuse a seat with evidence instead of detonating. + """ + if len(pairs) < 5: + return pairs, 1.0, 1.0 + A = np.array([np.asarray(p) for p, _ in pairs]) # parent + B = np.array([np.asarray(c) for _, c in pairs]) # child + + def _umeyama(src, dst): + mu_s, mu_d = src.mean(0), dst.mean(0) + Sc, Dc = src - mu_s, dst - mu_d + U, S, Vt = np.linalg.svd(Dc.T @ Sc / len(src)) + d = np.sign(np.linalg.det(U @ Vt)) + R = U @ np.diag([1.0, 1.0, d]) @ Vt + var = (Sc**2).sum() / len(src) + if var < 1e-12: + return None + s = (S * [1.0, 1.0, d]).sum() / var + return s, R, mu_d - s * R @ mu_s + + rng = np.random.default_rng(0) + thresh = max(1e-6, 0.10 * spread) + best_inl, best_count = None, -1 + for _ in range(256): + idx = rng.choice(len(pairs), 3, replace=False) + fit = _umeyama(A[idx], B[idx]) # parent -> child + if fit is None: + continue + s, R, t = fit + if not (1e-6 < s < 1e6): + continue + err = np.linalg.norm((s * (R @ A.T)).T + t - B, axis=1) + inl = err < thresh + if inl.sum() > best_count: + best_count, best_inl = int(inl.sum()), inl + if best_inl is None or best_count < 5: + return [], 1.0, 0.0 + fit = _umeyama(A[best_inl], B[best_inl]) + if fit is None: + return [], 1.0, 0.0 + s, R, t = fit + err = np.linalg.norm((s * (R @ A.T)).T + t - B, axis=1) + inl = err < thresh + inliers = [pairs[k] for k in range(len(pairs)) if inl[k]] + # pair_scale is child->parent (matches the solved aSb convention used by the scale band). + pair_scale = 1.0 / s if s > 1e-12 else float("inf") + return inliers, float(pair_scale), float(inl.mean()) + + +def _select_gid_point_correspondences( + parent_scene: GtsfmData, + child_scene: GtsfmData, + parent_index: tuple[np.ndarray, np.ndarray], + child_index: tuple[np.ndarray, np.ndarray], + max_correspondences: int = 100, +) -> list[tuple[np.ndarray, np.ndarray]]: + """Sim3 point correspondences by GLOBAL TRACK IDENTITY — no shared camera required. + + Both reconstructions cut their tracks from the same global track set, so matching by gid recognizes + the same physical point triangulated from DISJOINT camera sets (the cross-cluster edges the + shared-camera matcher cannot see). Pairs are sorted longest-tracks-first and capped. + """ + parent_by_gid: dict[int, int] = {} + for t_idx, track in enumerate(parent_scene.tracks()): + gid = _track_gid(track, parent_index) + if gid >= 0 and gid not in parent_by_gid: + parent_by_gid[gid] = t_idx + + pairs: list[tuple[int, int, int]] = [] # (combined_len, parent_idx, child_idx) + for c_idx, child_track in enumerate(child_scene.tracks()): + gid = _track_gid(child_track, child_index) + p_idx = parent_by_gid.get(gid) if gid >= 0 else None + if p_idx is not None: + combined = parent_scene.get_track(p_idx).numberMeasurements() + child_track.numberMeasurements() + pairs.append((combined, p_idx, c_idx)) + + pairs.sort(key=lambda p: -p[0]) + parent_tracks = parent_scene.tracks() + child_tracks = child_scene.tracks() + return [ + (parent_tracks[p].point3(), child_tracks[c].point3()) for _, p, c in pairs[:max_correspondences] + ] + + def _track_max_reprojection_error(scene: GtsfmData, track: gtsam.SfmTrack) -> float: errors, _ = compute_track_reprojection_errors(scene.cameras(), track) if np.isinf(errors).any(): return float("inf") finite_errors = errors[np.isfinite(errors)] + if finite_errors.size == 0: # all-NaN (e.g. every projection failed) — treat as unusable, not a crash + return float("inf") return float(np.max(finite_errors)) @@ -189,48 +421,144 @@ def merge_scenes_with_sim3_nonlinear( children_scenes: list[GtsfmData], max_track_correspondences_for_sim3: int = 150, scale_and_average_focal_length_in_merging: bool = False, + guard_child_min_cams: int = 30, + min_sim3_correspondences_large_child: int = 50, + sim3_scale_band: Tuple[float, float] = (0.0, float("inf")), ) -> GtsfmData: if len(children_scenes) == 0: return parent_scene - aTi_measurements = _create_unary_measurements(parent_scene) parent_camera_ids = set(parent_scene.get_valid_camera_indices()) - valid_child_scenes = [] + parent_gid_index = _scene_gid_index(parent_scene) + + # Parent camera-spread radius: the scene-scale yardstick for the point-pair RANSAC threshold and the + # point-factor sigma below. + _centers = np.array( + [np.array(parent_scene.get_camera(i).pose().translation()) for i in parent_camera_ids] + ) + spread = float(np.median(np.linalg.norm(_centers - _centers.mean(axis=0), axis=1))) if len(_centers) >= 2 else 1.0 + valid_child_scenes: list[GtsfmData] = [] + overlapping_points: list[list[tuple[np.ndarray, np.ndarray]]] = [] for i, child_scene in enumerate(children_scenes): child_camera_ids = set(child_scene.get_valid_camera_indices()) common_camera_ids = parent_camera_ids & child_camera_ids - if len(common_camera_ids) == 0: - logger.warning("Child scene %d has insufficient overlap with parent, skipping", i) + child_gid_index = _scene_gid_index(child_scene) + id_mode = parent_gid_index is not None and child_gid_index is not None + + if max_track_correspondences_for_sim3 > 0: + if id_mode: + # Match by GLOBAL TRACK IDENTITY: recognizes the same physical points triangulated from + # DISJOINT camera sets (the cross-cluster overlap the shared-camera matcher cannot see). + points = _select_gid_point_correspondences( + parent_scene, + child_scene, + parent_gid_index, + child_gid_index, + max_correspondences=max_track_correspondences_for_sim3, + ) + else: + points = _select_overlapping_track_point_correspondences( + parent_scene, + child_scene, + max_correspondences=max_track_correspondences_for_sim3, + preferred_min_track_length=3, + preferred_max_reproj_error_px=1.5, + ) + else: + points = [] + + # RANSAC-verify the pairs BEFORE the joint solve (ID mode): clean outlier matches, and measure the + # similarity the pair set actually implies. A coherently-wrong pair set (pair_scale far from 1) + # means the correspondences cannot be trusted to seat this child — refuse with evidence, pre-solve. + if id_mode and points: + points, pair_scale, pair_inlier_ratio = _prefilter_point_pairs(points, spread) + if points and not (sim3_scale_band[0] <= pair_scale <= sim3_scale_band[1]): + logger.warning( + "MERGE_GUARD: dropping child %d pre-solve (cams=%d, pairs imply scale=%.3g " + "[inlier_ratio=%.2f], shared_cams=%d) — inconsistent correspondences.", + i, + len(child_camera_ids), + pair_scale, + pair_inlier_ratio, + len(common_camera_ids), + ) + continue + + # MERGE_GUARD: a child with NO structure cannot be seated by anything — never merge it (a 36-cam + # leaf held by 1 track and 0-track children previously rode in on pose priors and detonated merges). + if child_scene.number_tracks() == 0: + logger.warning( + "MERGE_GUARD: dropping child %d (cams=%d, tracks=0) — structureless reconstruction.", + i, + len(child_camera_ids), + ) + continue + + # MERGE_GUARD (ID mode only, where correspondence count is a complete anchor measure): a large + # child without a reliable structural tie gets an under-constrained 7-DoF seat -> rigid stray + # block (ToL: 83 cams seated on 26 correspondences drifted 60-190m). Accuracy > camera count. + # OVERLAP ESCAPE: >=15 surviving shared cameras are a strong pose anchor on their own (measured: + # a 50-cam child with 21 shared cams seated fine at 25 correspondences; 7-9 shared cams failed) — + # BUT only for children with real structure: a 66-cam child holding ONE track escaped through the + # camera clause and detonated the root merge (1.6e22 px max). Pose anchors can place a block; they + # cannot vouch for a block whose own reconstruction is hollow. + camera_escape = len(common_camera_ids) >= 15 and child_scene.number_tracks() >= 50 + if ( + id_mode + and len(child_camera_ids) >= guard_child_min_cams + and len(points) < min_sim3_correspondences_large_child + and not camera_escape + ): + logger.warning( + "MERGE_GUARD: dropping child %d (cams=%d, id_correspondences=%d < %d, shared_cams=%d) — " + "under-constrained Sim3 seat.", + i, + len(child_camera_ids), + len(points), + min_sim3_correspondences_large_child, + len(common_camera_ids), + ) + continue + + # Admission needs SOME constraint: shared-camera pose factors or point correspondences. (With the + # gid index, a child sharing ZERO cameras can still be seated on its point correspondences alone — + # tree-level overlap that died in reconstruction no longer orphans an anchorable child.) + if len(common_camera_ids) == 0 and len(points) < 10: + logger.warning( + "Child scene %d has insufficient overlap with parent (0 shared cams, %d correspondences), skipping", + i, + len(points), + ) continue + valid_child_scenes.append(child_scene) + overlapping_points.append(points) + logger.info( + "Using %d parent-child point correspondences for child %d Sim3 alignment%s.", + len(points), + i, + " (global-track-id)" if id_mode else "", + ) if len(valid_child_scenes) == 0: return parent_scene aTi_measurements = _create_unary_measurements(parent_scene) bTi_measurements = [_create_unary_measurements(child_scene) for child_scene in valid_child_scenes] - if max_track_correspondences_for_sim3 > 0: - overlapping_points = [ - _select_overlapping_track_point_correspondences( - parent_scene, - child_scene, - max_correspondences=max_track_correspondences_for_sim3, - preferred_min_track_length=3, - preferred_max_reproj_error_px=1.5, - ) - for child_scene in valid_child_scenes - ] - else: - overlapping_points = [[] for _ in valid_child_scenes] - for child_idx, overlap_points in enumerate(overlapping_points): - logger.info( - "Using %d parent-child point correspondences for child %d Sim3 alignment.", len(overlap_points), child_idx - ) + + # Point-correspondence sigma. LEGACY (no gid sidecars) = fixed 1.0: points are effectively advisory + # and seats are pose-anchored — the exact R3-baseline semantics that produced the most complete ToL + # structure. ID MODE = 10% of the parent camera-spread radius (synthetic-validated stable incl. + # clumped/low-count pairs): identity-matched anchors carry real weight; the RANSAC prefilter above + # removes outlier pairs first. + point3_sigma = max(1e-4, 0.10 * spread) if parent_gid_index is not None else 1.0 try: # New GTSAM API supports adding parent-child 3D correspondences per child. - aligner = TrajectoryAlignerSim3(aTi_measurements, bTi_measurements, [], False, overlapping_points, 1.0) + aligner = TrajectoryAlignerSim3( + aTi_measurements, bTi_measurements, [], False, overlapping_points, point3_sigma + ) except TypeError: logger.warning( "TrajectoryAlignerSim3 point-correspondence API unavailable, falling back to pose-only alignment." @@ -246,6 +574,30 @@ def merge_scenes_with_sim3_nonlinear( opt_aSb = opt_bSa.inverse() opt_aSb_list.append(opt_aSb) + # MERGE_GUARD scale band: clusters are metric (global Fetzer focals), so a child's solved Sim3 scale + # must be ~1. A wild scale means the solve diverged (few/degenerate constraints — e.g. a 9-cam child + # on 12 correspondences detonated a merge to 5.7e8 px max reproj); merging it plants a garbage block + # that BA can only "fix" by deletion. Drop such children post-solve. + kept_children, kept_sim3 = [], [] + for i, (child_scene, opt_aSb) in enumerate(zip(valid_child_scenes, opt_aSb_list)): + s = float(opt_aSb.scale()) + if not (sim3_scale_band[0] <= s <= sim3_scale_band[1]): + logger.warning( + "MERGE_GUARD: dropping child %d post-solve (cams=%d, solved Sim3 scale=%.3g outside " + "[%g, %g]) — diverged seat.", + i, + len(child_scene.get_valid_camera_indices()), + s, + sim3_scale_band[0], + sim3_scale_band[1], + ) + continue + kept_children.append(child_scene) + kept_sim3.append(opt_aSb) + valid_child_scenes, opt_aSb_list = kept_children, kept_sim3 + if len(valid_child_scenes) == 0: + return parent_scene + merged = parent_scene for i, aTi in opt_aTi.items(): # Optionally scale-and-average intrinsics using child candidates in parent frame. @@ -314,6 +666,10 @@ class MergedNodeResult: pre_ba_scene: Optional[GtsfmData] metrics: GtsfmMetricsGroup pre_ba_metrics: GtsfmMetricsGroup + # Good-pose cameras dropped by the post-BA track filter at this node, keyed by camera index -> + # merged-frame Camera. Populated only at nodes where recover_trackless_cameras_in_retriangulation + # is set; the root node's set is what the post-merge retriangulation tries to geometrically recover. + trackless_cameras: Optional[dict] = None @dataclass(frozen=True) @@ -336,14 +692,17 @@ def annotate_scene_with_metadata( scene: Optional[GtsfmData], plots_dir: str | Path | None, cluster_label: Optional[str], + measurement_gid_index: Optional[tuple[np.ndarray, np.ndarray]] = None, ) -> Optional[GtsfmData]: - """Attach plotting metadata to a scene before it is merged downstream.""" + """Attach plotting metadata (and the optional per-cluster gid-index sidecar) to a scene before merging.""" if scene is None: return None if plots_dir is not None: setattr(scene, _SCENE_PLOTS_DIR_ATTR, Path(plots_dir)) if cluster_label is not None: setattr(scene, _SCENE_LABEL_ATTR, cluster_label) + if measurement_gid_index is not None: + setattr(scene, _SCENE_GID_INDEX_ATTR, measurement_gid_index) return scene @@ -612,6 +971,10 @@ def _run_export_task(payload: Tuple[Optional[Path], Future | MergedNodeResult]) gaussian_splats = merged_scene.get_gaussian_splats() if isinstance(gaussian_splats, GaussiansProtocol): try: + # Lazy import: cluster_anysplat transitively requires the GPU-only vggt package, which + # merging must not hard-depend on (only splat-carrying scenes ever reach this branch). + from gtsfm.cluster_optimizer.cluster_anysplat import save_splats + save_splats(merged_scene, merged_dir) except Exception as exc: logger.warning("⚠️ Failed to export Gaussian splats: %s", exc) @@ -624,6 +987,8 @@ def _run_export_task(payload: Tuple[Optional[Path], Future | MergedNodeResult]) gaussian_splats = pre_ba_merged.get_gaussian_splats() if isinstance(gaussian_splats, GaussiansProtocol): try: + from gtsfm.cluster_optimizer.cluster_anysplat import save_splats + save_splats(pre_ba_merged, pre_ba_dir) except Exception as exc: logger.warning("⚠️ Failed to export Gaussian splats: %s", exc) @@ -721,6 +1086,57 @@ def _drop_outlier_tracks(scene: GtsfmData) -> GtsfmData: return scene +def filter_tracks_by_triangulation_angle(scene: GtsfmData, min_angle_deg: float) -> GtsfmData: + """Drop tracks whose maximum pairwise triangulation angle is below ``min_angle_deg``. + + Low-parallax tracks are depth-unconstrained: the reprojection filters cannot see their depth + error (it lies along the viewing ray), so they survive BA as fuzz/spray around the structure. + Output-side filter only — intended for the final merged scene at export, never inside merging. + """ + if min_angle_deg <= 0: + return scene + cameras = scene.cameras() + centers = {cam_idx: np.asarray(camera.pose().translation(), dtype=float) for cam_idx, camera in cameras.items()} + + retained_tracks = [] + for track in scene.tracks(): + track_cams = {int(track.measurement(k)[0]) for k in range(track.numberMeasurements())} + cam_centers = np.array([centers[i] for i in track_cams if i in centers]) + if len(cam_centers) < 2: + continue # <2 posed views: no triangulation support at all + rays = cam_centers - np.asarray(track.point3(), dtype=float) + norms = np.linalg.norm(rays, axis=1) + rays = rays[norms > 1e-9] / norms[norms > 1e-9, None] + if len(rays) < 2: + continue + if len(rays) > 25: # cap the pairwise cost for long tracks (evenly spaced, deterministic) + rays = rays[np.linspace(0, len(rays) - 1, 25).astype(int)] + cos_matrix = np.clip(rays @ rays.T, -1.0, 1.0) + np.fill_diagonal(cos_matrix, 1.0) + if math.degrees(math.acos(float(cos_matrix.min()))) >= min_angle_deg: + retained_tracks.append(track) + + removed = scene.number_tracks() - len(retained_tracks) + logger.info( + "🔭 Triangulation-angle filter (<%.1f deg): dropped %d of %d tracks (%.1f%%).", + min_angle_deg, + removed, + scene.number_tracks(), + 100.0 * removed / max(scene.number_tracks(), 1), + ) + if removed == 0: + return scene + filtered = GtsfmData.from_cameras_and_tracks( + cameras, + retained_tracks, + number_images=scene.number_images(), + image_info=scene._clone_image_info(), + gaussian_splats=scene.get_gaussian_splats(), + ) + _propagate_scene_metadata(filtered, scene) + return filtered + + def _align_and_merge_results( result1: GtsfmData, result2: GtsfmData, @@ -778,6 +1194,10 @@ def combine_results( child_scenes: tuple[Optional[GtsfmData], ...] = tuple[GtsfmData | None, ...](child.scene for child in child_results) + # Union of the gid-index sidecars (this node's own + all children's) — used for gid-based duplicate + # fusion below and propagated on the outgoing scenes so ancestors can ID-match against this subtree. + node_gid_index = _union_gid_indices(_scene_gid_index(current), *[_scene_gid_index(c) for c in child_scenes]) + # Some stats for the merging metrics. parent_camera_set = set(current.get_valid_camera_indices()) if current is not None else set() child_camera_counts: list[int] = [] @@ -787,10 +1207,19 @@ def combine_results( child_camera_counts.append(len(child_cam_set)) child_camera_overlap_with_parent.append(len(child_cam_set & parent_camera_set)) - def _finalize_result(result_scene: Optional[GtsfmData], pre_ba_scene: Optional[GtsfmData]) -> MergedNodeResult: + def _finalize_result( + result_scene: Optional[GtsfmData], + pre_ba_scene: Optional[GtsfmData], + trackless_cameras: Optional[dict] = None, + ) -> MergedNodeResult: + # Sidecar rides the outgoing scenes (same rail as the plots/label attrs) — zero extra shipping. + for _scn in (result_scene, pre_ba_scene): + if _scn is not None and node_gid_index is not None: + setattr(_scn, _SCENE_GID_INDEX_ATTR, node_gid_index) return MergedNodeResult( scene=result_scene, pre_ba_scene=pre_ba_scene, + trackless_cameras=trackless_cameras, metrics=compute_merging_metrics( result_scene, cameras_gt=cameras_gt, @@ -838,6 +1267,9 @@ def _finalize_result(result_scene: Optional[GtsfmData], pre_ba_scene: Optional[G valid_child_scenes, max_track_correspondences_for_sim3=options.max_track_correspondences_for_sim3, scale_and_average_focal_length_in_merging=options.scale_and_average_focal_length, + guard_child_min_cams=options.guard_child_min_cams, + min_sim3_correspondences_large_child=options.min_sim3_correspondences_large_child, + sim3_scale_band=(options.sim3_scale_band_min, options.sim3_scale_band_max), ) _log_scene_reprojection_stats( merged, "Merged with children (nonlinear alignment)", plot_histograms=options.plot_reprojection_histograms @@ -864,6 +1296,7 @@ def _finalize_result(result_scene: Optional[GtsfmData], pre_ba_scene: Optional[G original_track_count = merged.number_tracks() merged_tracks: list = [] measurement_to_track: dict[tuple[int, int, int], int] = {} + gid_to_track: dict[int, int] = {} for track in merged.tracks(): measurements = [track.measurement(k) for k in range(track.numberMeasurements())] @@ -873,6 +1306,11 @@ def _finalize_result(result_scene: Optional[GtsfmData], pre_ba_scene: Optional[G if existing_idx is not None: target_idx = existing_idx break + # Same GLOBAL track triangulated by different clusters from DISJOINT cameras shares no + # measurement key — fuse it by global identity instead (kills ghost copies / double walls). + track_gid = _track_gid(track, node_gid_index) + if target_idx is None and track_gid >= 0: + target_idx = gid_to_track.get(track_gid) if target_idx is None: merged_tracks.append(track) @@ -890,6 +1328,8 @@ def _finalize_result(result_scene: Optional[GtsfmData], pre_ba_scene: Optional[G for m_idx in range(base_track.numberMeasurements()): cam_idx, uv = base_track.measurement(m_idx) measurement_to_track[_measurement_key(cam_idx, uv)] = target_idx + if track_gid >= 0: + gid_to_track[track_gid] = target_idx if len(merged_tracks) < original_track_count: logger.info( @@ -940,12 +1380,24 @@ def _finalize_result(result_scene: Optional[GtsfmData], pre_ba_scene: Optional[G plot_histograms=options.plot_reprojection_histograms, ) + trackless_cameras: Optional[dict[int, gtsfm_types.CAMERA_TYPE]] = None if options.allow_post_ba_reproj_filtering: + scene_before_filter = merged_with_ba + cams_before_filter = set(scene_before_filter.get_valid_camera_indices()) merged_with_ba = merged_with_ba.filter_landmark_measurements( options.post_ba_max_reproj_error, options.min_track_length, retain_cameras_without_tracks=options.keep_all_cameras, ) + if options.recover_trackless_cameras_in_retriangulation: + # Capture the good-pose cameras the track filter just dropped (e.g. low VGGT-depth-conf + # cams) with their merged-frame poses, so the post-merge retriangulation can recover them. + dropped_ids = cams_before_filter - set(merged_with_ba.get_valid_camera_indices()) + trackless_cameras = { + i: scene_before_filter.get_camera(i) + for i in dropped_ids + if scene_before_filter.get_camera(i) is not None + } _log_scene_reprojection_stats( merged_with_ba, "merged result (with ba + outlier filtering)", @@ -978,15 +1430,15 @@ def _finalize_result(result_scene: Optional[GtsfmData], pre_ba_scene: Optional[G except Exception as e: logger.warning("⚠️ Failed to align and merge gaussians: %s", e) merged_with_ba.set_gaussian_splats(merged_gaussians) - return _finalize_result(merged_with_ba, merged) + return _finalize_result(merged_with_ba, merged, trackless_cameras) except Exception as alignment_exc: logger.warning("⚠️ Failed to compute pre/post BA Sim(3): %s", alignment_exc) - return _finalize_result(merged_with_ba, merged) + return _finalize_result(merged_with_ba, merged, trackless_cameras) else: logger.info("✖️ No Gaussians to merge") - return _finalize_result(merged_with_ba, merged) + return _finalize_result(merged_with_ba, merged, trackless_cameras) except Exception as exc: logger.warning("⚠️ Failed to run bundle adjustment: %s", exc) return _finalize_result(merged, None) diff --git a/gtsfm/cluster_optimizer/__init__.py b/gtsfm/cluster_optimizer/__init__.py index 36338c4da..2c29d8c31 100644 --- a/gtsfm/cluster_optimizer/__init__.py +++ b/gtsfm/cluster_optimizer/__init__.py @@ -13,6 +13,7 @@ "Anysplat", "Cacher", "VggtWithFrontend", + "VggtOmegaWithFrontend", "logger", "save_metrics_reports", ] @@ -20,11 +21,12 @@ # Provide symbols to type checkers/IDEs without incurring runtime imports. if TYPE_CHECKING: from .cluster_anysplat import ClusterAnySplat as Anysplat + from .cluster_fast_vggt import ClusterFastVGGT as FastVggt from .cluster_mvo import ClusterMVO as Multiview from .cluster_optimizer_base import ClusterOptimizerBase as Base from .cluster_optimizer_cacher import ClusterOptimizerCacher as Cacher from .cluster_vggt import ClusterVGGT as Vggt - from .cluster_fast_vggt import ClusterFastVGGT as FastVggt + from .cluster_vggt_omega_with_frontend import ClusterVGGTOmegaWithFrontend as VggtOmegaWithFrontend from .cluster_vggt_with_frontend import ClusterVGGTWithFrontend as VggtWithFrontend # Short name → (module, class) for lazy attribute access. @@ -36,6 +38,10 @@ "Anysplat": ("gtsfm.cluster_optimizer.cluster_anysplat", "ClusterAnySplat"), "Cacher": ("gtsfm.cluster_optimizer.cluster_optimizer_cacher", "ClusterOptimizerCacher"), "VggtWithFrontend": ("gtsfm.cluster_optimizer.cluster_vggt_with_frontend", "ClusterVGGTWithFrontend"), + "VggtOmegaWithFrontend": ( + "gtsfm.cluster_optimizer.cluster_vggt_omega_with_frontend", + "ClusterVGGTOmegaWithFrontend", + ), } diff --git a/gtsfm/cluster_optimizer/cluster_mvo.py b/gtsfm/cluster_optimizer/cluster_mvo.py index e243e8c71..71ddd912c 100644 --- a/gtsfm/cluster_optimizer/cluster_mvo.py +++ b/gtsfm/cluster_optimizer/cluster_mvo.py @@ -6,11 +6,10 @@ import time from dataclasses import dataclass from pathlib import Path -from typing import Any, Mapping, Optional, cast +from typing import Any, Mapping, Optional import numpy as np from dask.delayed import Delayed, delayed -from dask.distributed import Future, worker_client from gtsam import Pose3, Similarity3 # type: ignore import gtsfm.common.types as gtsfm_types @@ -33,7 +32,11 @@ from gtsfm.products.one_view_data import OneViewData from gtsfm.products.two_view_result import TwoViewResult from gtsfm.products.visibility_graph import AnnotatedGraph, VisibilityGraph -from gtsfm.two_view_estimator import TwoViewEstimator, create_two_view_estimator_futures +from gtsfm.two_view_estimator import ( + TwoViewEstimator, + create_two_view_results_inline, + create_v_corr_idxs_inline, +) from gtsfm.ui.gtsfm_process import UiMetadata from gtsfm.utils import transform @@ -114,22 +117,25 @@ def get_ui_metadata() -> UiMetadata: def _run_correspondence_generator( correspondence_generator: CorrespondenceGeneratorBase, visibility_graph: VisibilityGraph, - image_future_keys: list[str], + images: list[Image], ) -> tuple[list[Keypoints], AnnotatedGraph[np.ndarray], float]: - """Execute correspondence generation inside a worker task.""" + """Run correspondence generation inline (no nested Dask submission). + + ``images`` arrives materialized as a normal Dask dependency (``images[i]`` is image ``i``). + Detection + matching run in plain loops over the cache-backed primitives — no ``worker_client()``, + so the entire frontend working set is never held resident on one worker at once. + """ logger.info("🔵 Running correspondence generation for %d pairs.", len(visibility_graph)) if len(visibility_graph) == 0: return [], {}, 0.0 - with worker_client() as nested_client: - image_futures = [Future(key=key, client=nested_client) for key in image_future_keys] - start_time = time.time() - keypoints_list, putative_corr_idxs_dict = correspondence_generator.generate_correspondences( - nested_client, image_futures, visibility_graph - ) - duration_sec = time.time() - start_time + start_time = time.time() + keypoints_list, putative_corr_idxs_dict = correspondence_generator.generate_correspondences_inline( + images, visibility_graph + ) + duration_sec = time.time() - start_time return keypoints_list, putative_corr_idxs_dict, duration_sec @@ -142,24 +148,20 @@ def _run_two_view_estimation( gt_scene_mesh: Optional[Any], one_view_data_dict: dict[int, OneViewData], ) -> tuple[AnnotatedGraph[TwoViewResult], float]: - """Execute two-view estimation inside a worker task.""" + """Run two-view estimation inline (no nested Dask submission).""" logger.info("🔵 Running two-view estimation for %d pairs.", len(putative_corr_idxs_dict)) - with worker_client() as nested_client: - start_time = time.time() - two_view_result_futures = create_two_view_estimator_futures( - client=nested_client, - two_view_estimator=two_view_estimator, - keypoints_list=keypoints_list, - putative_corr_idxs_dict=putative_corr_idxs_dict, - relative_pose_priors=relative_pose_priors, - gt_scene_mesh=gt_scene_mesh, - one_view_data_dict=one_view_data_dict, - ) - gathered_tve_futures = nested_client.gather(two_view_result_futures) - duration_sec = time.time() - start_time + start_time = time.time() + all_two_view_results = create_two_view_results_inline( + two_view_estimator=two_view_estimator, + keypoints_list=keypoints_list, + putative_corr_idxs_dict=putative_corr_idxs_dict, + relative_pose_priors=relative_pose_priors, + gt_scene_mesh=gt_scene_mesh, + one_view_data_dict=one_view_data_dict, + ) + duration_sec = time.time() - start_time - all_two_view_results = cast(AnnotatedGraph[TwoViewResult], gathered_tve_futures) valid_two_view_results = {edge: result for edge, result in all_two_view_results.items() if result.valid()} n_total = len(all_two_view_results) @@ -171,6 +173,46 @@ def _run_two_view_estimation( return valid_two_view_results, duration_sec + @staticmethod + def _run_two_view_v_corr_idxs( + two_view_estimator: TwoViewEstimator, + keypoints_list: list[Keypoints], + putative_corr_idxs_dict: AnnotatedGraph[np.ndarray], + relative_pose_priors: AnnotatedGraph[PosePrior], + gt_scene_mesh: Optional[Any], + one_view_data_dict: dict[int, OneViewData], + ) -> tuple[AnnotatedGraph[np.ndarray], float]: + """Streaming two-view estimation returning only valid edges' v_corr_idxs (memory-bounded). + + Same computation as ``_run_two_view_estimation`` but keeps only the verified-correspondence + indices, dropping each heavy ``TwoViewResult`` as it goes so the worker never accumulates all + results. Used by the global verified-pipeline pass, where only ``v_corr_idxs`` is consumed + (relative poses come from VGGT per cluster). run_2view side effects (cacher/DB) are unchanged. + """ + logger.info( + "🔵 Running streaming two-view estimation for %d pairs (v_corr_idxs only).", + len(putative_corr_idxs_dict), + ) + + start_time = time.time() + v_corr_idxs_dict = create_v_corr_idxs_inline( + two_view_estimator=two_view_estimator, + keypoints_list=keypoints_list, + putative_corr_idxs_dict=putative_corr_idxs_dict, + relative_pose_priors=relative_pose_priors, + gt_scene_mesh=gt_scene_mesh, + one_view_data_dict=one_view_data_dict, + ) + duration_sec = time.time() - start_time + + logger.info( + "Streaming two-view estimation: %d/%d pairs valid.", + len(v_corr_idxs_dict), + len(putative_corr_idxs_dict), + ) + + return v_corr_idxs_dict, duration_sec + @staticmethod def _save_two_view_visualizations( images_by_index: dict[int, Image], @@ -202,14 +244,16 @@ def _build_frontend_graphs(self, context: ClusterContext) -> FrontendGraphs: """Create delayed nodes for the full front-end pipeline.""" visibility_edges = list(context.visibility_graph) - image_future_keys = [context.image_future_map[idx].key for idx in range(context.num_images)] + # Pass the image futures directly as task inputs; Dask materializes them as a normal dependency + # (no nested gather), so the correspondence task receives concrete images[i] in index order. + image_futures = [context.image_future_map[idx] for idx in range(context.num_images)] keypoints_graph, putative_corr_idxs_graph, correspondence_duration_graph = delayed( ClusterMVO._run_correspondence_generator, nout=3 )( self.correspondence_generator, visibility_edges, - image_future_keys, + image_futures, ) padded_keypoints_graph = delayed(_pad_keypoints_list)(keypoints_graph, context.num_images) diff --git a/gtsfm/cluster_optimizer/cluster_optimizer_base.py b/gtsfm/cluster_optimizer/cluster_optimizer_base.py index 3d762c96e..338ae2c6a 100644 --- a/gtsfm/cluster_optimizer/cluster_optimizer_base.py +++ b/gtsfm/cluster_optimizer/cluster_optimizer_base.py @@ -49,6 +49,19 @@ class ClusterContext: cluster_path: tuple[int, ...] label: str visibility_graph: VisibilityGraph + # Global Fetzer focals (cam idx -> calibration), computed once over the full verified view graph. + # Used by ClusterVGGTWithFrontend when use_global_view_graph_calibration is set, in place of the + # per-cluster calibration (which falls back to VGGT focals for cameras with few in-cluster F-edges). + global_refined_intrinsics: dict | None = None + # Global frontend products from the verified two-view pass, reused per cluster instead of re-running + # the per-cluster correspondence generation. v_corr keyed by image-pair; keypoints indexed by image. + # Used by ClusterVGGTWithFrontend when reuse_global_correspondences is set. See ClusterContext plumbing. + global_v_corr_idxs_dict: dict | None = None + global_keypoints: list | None = None + # Per-cluster slice of the packed (camera, pixel) -> global-track-id index (sorted int64 keys, int32 + # gids). Attached to this cluster's reconstruction as a sidecar so merges can match tracks by GLOBAL + # identity across clusters that share no cameras. None disables ID-matching (non-verified pipelines). + measurement_gid_index: tuple | None = None @property def is_root(self) -> bool: diff --git a/gtsfm/cluster_optimizer/cluster_optimizer_cacher.py b/gtsfm/cluster_optimizer/cluster_optimizer_cacher.py index b3fede564..e2668cfeb 100644 --- a/gtsfm/cluster_optimizer/cluster_optimizer_cacher.py +++ b/gtsfm/cluster_optimizer/cluster_optimizer_cacher.py @@ -27,7 +27,7 @@ from gtsfm.products.one_view_data import OneViewData # Keep cache location consistent with other cachers. -CACHE_ROOT_PATH = Path(__file__).resolve().parent.parent.parent / "cache" +CACHE_ROOT_PATH = cache_utils.get_cache_root() # honors GTSFM_CACHE_ROOT env var logger = logger_utils.get_logger() diff --git a/gtsfm/cluster_optimizer/cluster_vggt.py b/gtsfm/cluster_optimizer/cluster_vggt.py index 808a0f279..5914d44bc 100644 --- a/gtsfm/cluster_optimizer/cluster_vggt.py +++ b/gtsfm/cluster_optimizer/cluster_vggt.py @@ -53,6 +53,7 @@ def _run_cluster_ba( cluster_label: Optional[str] = None, tracks_2d: Optional[list[SfmTrack2d]] = None, use_multi_view_retriangulation: bool = False, + run_bundle_adjustment: bool = True, ) -> tuple[GtsfmData, GtsfmData]: """Run cluster-level BA on a GtsfmData result. @@ -92,6 +93,21 @@ def _run_cluster_ba( if not should_run_ba: return gtsfm_data, pre_ba_data + if not run_bundle_adjustment: + # ToL census (2026-07-06): per-cluster BA degraded 7/8 GT-gradeable clusters (median 2.90->4.34m + # vs GT) — the ~4px pose/Fetzer-K reprojection inconsistency gets resolved by moving the FREE + # poses instead of the pinned focals. Raw VGGT poses + global-Fetzer K are kept verbatim + # (bit-identical intrinsics per camera across clusters, which is what the Sim3 merges need); + # the pose-pinned merge BA downstream does the structure polishing safely. The 3px post-BA + # filter is NOT applied here: unpolished structure sits at ~4px and the merge pre-filter (14px) + # is the gate that admits it. + logger.info( + "%s🛑 Per-cluster BA disabled: keeping raw geometry (%d tracks after pre-BA filter).", + f"[{cluster_label}] " if cluster_label else "", + gtsfm_data.number_tracks(), + ) + return gtsfm_data, pre_ba_data + try: optimizer = ba_options.to_optimizer(min_track_length=min_track_length) gtsfm_data_with_ba, _ = optimizer.run_simple_ba(gtsfm_data) @@ -136,12 +152,22 @@ def _load_vggt_inputs( indices: list[int], mode: str, *, + transformer=None, save_processed_image: bool = False, output_root: Optional[str] = None, image_names: Optional[tuple[str, ...]] = None, ): - """Load and preprocess a batch of images for VGGT.""" - image_batch, original_coords = load_image_batch_vggt_loader(loader, indices, mode=mode) + """Load and preprocess a batch of images for the geometry model. + + Preprocessing follows the geometry transformer: VGGT and VGGT-Omega differ in resolution / patch + alignment / cropping, and the per-pixel depth lookup downstream indexes the model's depth map using + `original_coords` from here — so the loader must match the model. `transformer=None` keeps the legacy + VGGT loader (back-compat). + """ + if transformer is not None: + image_batch, original_coords = transformer.load_image_batch(loader, indices, mode=mode) + else: + image_batch, original_coords = load_image_batch_vggt_loader(loader, indices, mode=mode) if not save_processed_image or output_root is None or image_names is None: return image_batch, original_coords if len(image_names) != image_batch.shape[0]: @@ -487,6 +513,7 @@ def create_computation_graph( context.loader, global_indices, mode=self._input_mode, + transformer=self.geometry_transformer, save_processed_image=self._save_processed_image, output_root=str(context.output_paths.results), image_names=image_names, diff --git a/gtsfm/cluster_optimizer/cluster_vggt_omega_with_frontend.py b/gtsfm/cluster_optimizer/cluster_vggt_omega_with_frontend.py new file mode 100644 index 000000000..10a28a8f6 --- /dev/null +++ b/gtsfm/cluster_optimizer/cluster_vggt_omega_with_frontend.py @@ -0,0 +1,404 @@ +"""Cluster optimizer that combines a traditional MVO frontend with VGGT-Omega geometry + BA.""" + +from __future__ import annotations + +from typing import Any, Hashable, NamedTuple, Optional + +import gtsam +import numpy as np +import torch +from dask.delayed import delayed +from gtsam import Point2, Point3, SfmTrack + +import gtsfm.common.types as gtsfm_types +from gtsfm.bundle.bundle_adjustment import BundleAdjustmentOptions +from gtsfm.cluster_optimizer.cluster_mvo import ClusterMVO +from gtsfm.cluster_optimizer.cluster_optimizer_base import ClusterComputationGraph, ClusterContext +from gtsfm.cluster_optimizer.cluster_vggt import ( + _aggregate_vggt_metrics, + _run_cluster_ba, + _save_pre_ba_reconstruction_as_text, + _save_reconstruction_as_text, +) +from gtsfm.common.gtsfm_data import GtsfmData +from gtsfm.common.sfm_track import SfmTrack2d +from gtsfm.frontend.correspondence_generator.correspondence_generator_base import CorrespondenceGeneratorBase +from gtsfm.frontend.vggt_omega_geometry_transformer import ( + VggtOmegaGeometryTransformer, + load_image_batch_vggt_omega_loader, + load_model, +) +from gtsfm.multi_view_optimizer import get_2d_tracks +from gtsfm.products.visibility_graph import visibility_graph_keys +from gtsfm.two_view_estimator import TwoViewEstimator +from gtsfm.ui.gtsfm_process import UiMetadata +from gtsfm.utils import torch as torch_utils +from gtsfm.utils.logger import get_logger + +logger = get_logger() + +# Module-level cache to avoid reloading VGGT weights per cluster. +_VGGT_OMEGA_MODEL_CACHE: dict[Hashable, Any] = {} + + +class VggtOmegaGeometryResult(NamedTuple): + """Outputs of VGGT geometry prediction needed for downstream processing.""" + + cameras: dict[int, gtsfm_types.CAMERA_TYPE] + dense_points: np.ndarray # (N, H, W, 3) world-space 3D points per pixel + depth_confidence: np.ndarray # (N, H, W) per-pixel depth confidence scores + original_coords: np.ndarray # (N, 6) VGGT crop/pad metadata + + +def _extract_v_corr_idxs(two_view_results) -> dict: + """Pull verified correspondence index arrays out of two-view results.""" + return {ij: result.v_corr_idxs for ij, result in two_view_results.items()} + + +def _get_image_shapes(loader, image_indices: tuple[int, ...]) -> dict[int, tuple[int, int]]: + """Return original (height, width) for each requested image index.""" + return {idx: loader.get_image(idx).value_array.shape[:2] for idx in image_indices} + + +def _resolve_vggt_omega_model(cache_key: Hashable | None) -> Any | None: + """Fetch (or lazily load) a VGGT Omega model for the current worker.""" + if cache_key is None: + return None + if cache_key in _VGGT_OMEGA_MODEL_CACHE: + return _VGGT_OMEGA_MODEL_CACHE[cache_key] + logger.info("⏳ Loading VGGT Omega model weights...") + model = load_model(torch.device("cuda")) + _VGGT_OMEGA_MODEL_CACHE[cache_key] = model + logger.info("✅ VGGT Omega model weights loaded successfully.") + return model + + +def _run_vggt_omega_geometry( + image_batch: torch.Tensor, + original_coords: torch.Tensor, + *, + transformer: VggtOmegaGeometryTransformer, + image_indices: tuple[int, ...], + seed: int = 42, + model_cache_key: Hashable | None = None, + cluster_label: Optional[str] = None, +) -> VggtOmegaGeometryResult: + """Run VGGT Omega geometry prediction, returning cameras and dense 3D points.""" + torch.manual_seed(seed) + np.random.seed(seed) + if torch.cuda.is_available(): + torch.cuda.manual_seed_all(seed) + + if cluster_label: + logger.info("🔵 Running VGGT geometry on %s with %d images.", cluster_label, image_batch.shape[0]) + + cached_model = _resolve_vggt_omega_model(model_cache_key) + geo_output = transformer.predict(image_batch, model=cached_model) + + original_coords_np = original_coords.to(torch.float32).cpu().numpy() + cameras = { + global_idx: torch_utils.camera_from_matrices( + geo_output.extrinsic[local_idx].to(torch.float32).cpu().numpy(), + geo_output.intrinsic[local_idx].to(torch.float32).cpu().numpy(), + crop_coords=original_coords_np[local_idx], + use_cal3ds2=True, + ) + for local_idx, global_idx in enumerate(image_indices) + } + + result = VggtOmegaGeometryResult( + cameras=cameras, + dense_points=geo_output.dense_points.to(torch.float32).cpu().numpy(), + depth_confidence=geo_output.depth_confidence.to(torch.float32).cpu().numpy(), + original_coords=original_coords_np, + ) + + if geo_output.device.type == "cuda": + del geo_output + torch.cuda.empty_cache() + + return result + + +def _scale_camera_intrinsics( + camera: gtsfm_types.CAMERA_TYPE, scale_x: float, scale_y: float +) -> gtsfm_types.CAMERA_TYPE: + """Return a copy of camera with intrinsics uniformly scaled, pose unchanged.""" + pose = camera.pose() + cal = camera.calibration() + if isinstance(cal, gtsam.Cal3DS2): + return gtsam.PinholeCameraCal3DS2( + pose, + gtsam.Cal3DS2( + cal.fx() * scale_x, + cal.fy() * scale_y, + 0.0, + cal.px() * scale_x, + cal.py() * scale_y, + cal.k1(), + cal.k2(), + cal.p1(), + cal.p2(), + ), + ) + if isinstance(cal, gtsam.Cal3_S2): + return gtsam.PinholeCameraCal3_S2( + pose, + gtsam.Cal3_S2(cal.fx() * scale_x, cal.fy() * scale_y, 0.0, cal.px() * scale_x, cal.py() * scale_y), + ) + raise ValueError(f"Unsupported calibration type: {type(cal)}") + + +def _build_gtsfm_data_from_vggt_omega_depth( + vggt_omega_result: VggtOmegaGeometryResult, + tracks_2d: list[SfmTrack2d], + image_shapes: dict[int, tuple[int, int]], + image_indices: tuple[int, ...], + num_images: int, + min_track_length: int = 2, +) -> GtsfmData: + """Build GtsfmData using VGGT Omega cameras (rescaled to original resolution) and frontend 2D tracks. + + VGGT Omega camera intrinsics are scaled from VGGT pixel space to original image resolution so + that the frontend keypoints (in original coords) can be used directly as BA measurements. + VGGT Omega pixel coordinates are used only to look up per-pixel depth values for 3D initialisation. + + Args: + vggt_omega_result: Cameras, dense 3D points, and crop metadata from VGGT. + tracks_2d: 2D feature tracks from the frontend (in original image coordinates). + image_shapes: Original (height, width) per global image index. + image_indices: Ordered global image indices corresponding to the N VGGT frames. + num_images: Total number of images in the scene. + min_track_length: Minimum observations per track; shorter tracks are dropped. + + Returns: + GtsfmData with intrinsics-rescaled VGGT cameras and depth-initialised tracks whose + 2D measurements are in the original keypoint coordinate system. + """ + cameras = vggt_omega_result.cameras + dense_points = vggt_omega_result.dense_points # (N, H_vggt, W_vggt, 3) + depth_confidence = vggt_omega_result.depth_confidence # (N, H_vggt, W_vggt) + original_coords = vggt_omega_result.original_coords # (N, 6): [left, top, right, bottom, sw, sh] + + _, H_vggt, W_vggt = dense_points.shape[:3] + global_to_local = {gidx: lidx for lidx, gidx in enumerate(image_indices)} + + # Register cameras with intrinsics rescaled to original image resolution. + gtsfm_data = GtsfmData(number_images=num_images) + for global_idx, camera in cameras.items(): + if global_idx in image_shapes and global_idx in global_to_local: + orig_H, orig_W = image_shapes[global_idx] + local_idx = global_to_local[global_idx] + scaled_W = float(original_coords[local_idx, 4]) + scaled_H = float(original_coords[local_idx, 5]) + camera = _scale_camera_intrinsics(camera, scale_x=orig_W / scaled_W, scale_y=orig_H / scaled_H) + gtsfm_data.add_camera(global_idx, camera) + + for track_2d in tracks_2d: + if track_2d.number_measurements() < min_track_length: + continue + + points_3d: list[np.ndarray] = [] + confidences: list[float] = [] + valid_measurements: list[tuple[int, np.ndarray]] = [] + + for m in track_2d.measurements: + global_idx = m.i + if global_idx not in global_to_local or global_idx not in image_shapes: + continue + + local_idx = global_to_local[global_idx] + orig_H, orig_W = image_shapes[global_idx] + + # Map frontend keypoints into VGGT Omega dense-map coordinates using the actual + # per-axis resized image dimensions plus the crop/pad offsets recorded in + # original_coords for this image. + left, top = original_coords[local_idx, 0], original_coords[local_idx, 1] + scaled_W, scaled_H = original_coords[local_idx, 4], original_coords[local_idx, 5] + u_scale = scaled_W / orig_W if orig_W > 0 else 0.0 + v_scale = scaled_H / orig_H if orig_H > 0 else 0.0 + u_c = int(round(m.uv[0] * u_scale - left)) + v_c = int(round(m.uv[1] * v_scale - top)) + if not (0.0 <= u_c < W_vggt and 0.0 <= v_c < H_vggt): + continue + + pt3d = dense_points[local_idx, v_c, u_c] + conf = float(depth_confidence[local_idx, v_c, u_c]) + if np.isfinite(pt3d).all() and np.isfinite(conf): + points_3d.append(pt3d) + confidences.append(conf) + valid_measurements.append((global_idx, m.uv)) # original keypoint coords + + if len(points_3d) < min_track_length: + continue + + weights = np.array(confidences) + weights /= weights.sum() + point_3d_mean = np.average(points_3d, axis=0, weights=weights) + sfm_track = SfmTrack(Point3(*point_3d_mean.astype(float))) + for gidx, uv in valid_measurements: + if gidx in cameras: + sfm_track.addMeasurement(gidx, Point2(*uv.astype(float))) + + if sfm_track.numberMeasurements() >= min_track_length: + gtsfm_data.add_track(sfm_track) + + logger.info("Built GtsfmData with %d cameras and %d tracks.", len(cameras), gtsfm_data.number_tracks()) + return gtsfm_data + + +def _load_vggt_omega_inputs( + loader, + indices: list[int], +): + """Load and preprocess a batch of images for VGGT Omega.""" + image_batch, original_coords = load_image_batch_vggt_omega_loader(loader, indices) + return image_batch, original_coords + + +class ClusterVGGTOmegaWithFrontend(ClusterMVO): + """Cluster optimizer that combines a traditional MVO frontend with VGGT poses. + + Camera poses come from VGGT Omega geometry prediction. 2D feature tracks come from the + frontend (correspondence generation + two-view estimation). Each track's 3D point is + initialised from VGGT Omega's dense depth map rather than via triangulation, then refined + by cluster-level bundle adjustment. + """ + + def __init__( + self, + correspondence_generator: CorrespondenceGeneratorBase, + two_view_estimator: TwoViewEstimator, + geometry_transformer: VggtOmegaGeometryTransformer | None = None, + ba_options: BundleAdjustmentOptions | None = None, + # Cluster BA params + pre_ba_max_reproj_error: float = 14.0, + post_ba_max_reproj_error: float = 3.0, + drop_camera_with_no_track: bool = False, + min_track_length: int = 2, + # VGGT model loading + seed: int = 42, + model_cache_key: Hashable | bool | None = None, + metric_constructed_only: bool = False, + # Frontend params + save_two_view_viz: bool = False, + pose_angular_error_thresh: float = 3, + output_worker: Optional[str] = None, + ) -> None: + super().__init__( + correspondence_generator=correspondence_generator, + two_view_estimator=two_view_estimator, + multiview_optimizer=None, # VGGT Omega depth replaces triangulation + MVO + save_two_view_viz=save_two_view_viz, + pose_angular_error_thresh=pose_angular_error_thresh, + output_worker=output_worker, + ) + self.geometry_transformer = geometry_transformer or VggtOmegaGeometryTransformer() + self.ba_options = ba_options or BundleAdjustmentOptions() + self._pre_ba_max_reproj_error = pre_ba_max_reproj_error + self._post_ba_max_reproj_error = post_ba_max_reproj_error + self._drop_camera_with_no_track = drop_camera_with_no_track + self._min_track_length = min_track_length + self._metric_constructed_only = metric_constructed_only + self._seed = seed + + if model_cache_key is False: + self._model_cache_key: Hashable | None = None + elif model_cache_key is None: + self._model_cache_key = "default_vggt_omega_loader" + else: + self._model_cache_key = model_cache_key + + def __repr__(self) -> str: + components = [ + f"correspondence_generator={self.correspondence_generator}", + f"two_view_estimator={self.two_view_estimator}", + f"ba_options={self.ba_options}", + ] + return "ClusterVGGTOmegaWithFrontend(\n " + ",\n ".join(components) + "\n)" + + @staticmethod + def get_ui_metadata() -> UiMetadata: + return UiMetadata( + display_name="VGGT Omega + Frontend", + input_products=("Key Images",), + output_products=("VGGT Omega Reconstruction",), + parent_plate="Cluster Optimizer", + ) + + def create_computation_graph(self, context: ClusterContext) -> ClusterComputationGraph | None: + keys = sorted(visibility_graph_keys(context.visibility_graph)) + if not keys: + return None + + global_indices = tuple(int(idx) for idx in keys) + + # Traditional frontend. + frontend_graphs = self._build_frontend_graphs(context) + io_tasks, metrics = self._build_frontend_output_graphs(context, frontend_graphs) + + # VGGT Omega geometry prediction → cameras + dense 3D points. + image_batch_graph, original_coords_graph = delayed(_load_vggt_omega_inputs, nout=2)( + context.loader, + global_indices, + ) + vggt_omega_result_graph = delayed(_run_vggt_omega_geometry)( + image_batch_graph, + original_coords_graph, + transformer=self.geometry_transformer, + image_indices=global_indices, + seed=self._seed, + model_cache_key=self._model_cache_key, + cluster_label=context.label, + ) + + # 3. 2D tracks from frontend correspondences. + v_corr_idxs_graph = delayed(_extract_v_corr_idxs)(frontend_graphs.two_view_results) + tracks_2d_graph = delayed(get_2d_tracks)(v_corr_idxs_graph, frontend_graphs.padded_keypoints) + + # 4. Original image shapes (needed to map frontend pixel coords → VGGT Omega pixel coords). + image_shapes_graph = delayed(_get_image_shapes)(context.loader, global_indices) + + # 5. Build GtsfmData: lift 2D tracks to 3D using VGGT depth map. + ba_input_graph = delayed(_build_gtsfm_data_from_vggt_omega_depth)( + vggt_omega_result_graph, + tracks_2d_graph, + image_shapes=image_shapes_graph, + image_indices=global_indices, + num_images=context.num_images, + min_track_length=self._min_track_length, + ) + + # 6. Cluster-level BA. + ba_result_graph, pre_ba_result_graph = delayed(_run_cluster_ba, nout=2)( + ba_input_graph, + ba_options=self.ba_options, + pre_ba_max_reproj_error=self._pre_ba_max_reproj_error, + post_ba_max_reproj_error=self._post_ba_max_reproj_error, + drop_camera_with_no_track=self._drop_camera_with_no_track, + min_track_length=self._min_track_length, + cluster_label=context.label, + ) + + # 7. Metrics + I/O. + cameras_gt = [context.one_view_data_dict[idx].camera_gt for idx in range(context.num_images)] + metrics.append( + delayed(_aggregate_vggt_metrics)( + ba_result_graph, + cameras_gt=cameras_gt, + pre_ba_result=pre_ba_result_graph, + save_dir=str(context.output_paths.metrics), + metric_constructed_only=self._metric_constructed_only, + ) + ) + with self._output_annotation(): + io_tasks.append(delayed(_save_reconstruction_as_text)(ba_result_graph, context.output_paths.results)) + io_tasks.append( + delayed(_save_pre_ba_reconstruction_as_text)(pre_ba_result_graph, context.output_paths.results) + ) + + return ClusterComputationGraph( + io_tasks=tuple(io_tasks), + metric_tasks=tuple(metrics), + sfm_result=ba_result_graph, + ) diff --git a/gtsfm/cluster_optimizer/cluster_vggt_with_frontend.py b/gtsfm/cluster_optimizer/cluster_vggt_with_frontend.py index 030881f43..b87645b90 100644 --- a/gtsfm/cluster_optimizer/cluster_vggt_with_frontend.py +++ b/gtsfm/cluster_optimizer/cluster_vggt_with_frontend.py @@ -11,7 +11,7 @@ from gtsam import Point2, Point3, SfmTrack import gtsfm.common.types as gtsfm_types -from gtsfm.bundle.bundle_adjustment import BundleAdjustmentOptions +from gtsfm.bundle.bundle_adjustment import BundleAdjustmentOptions, multi_view_retriangulate_from_2d_tracks from gtsfm.cluster_optimizer.cluster_mvo import ClusterMVO from gtsfm.cluster_optimizer.cluster_optimizer_base import ClusterComputationGraph, ClusterContext from gtsfm.cluster_optimizer.cluster_vggt import ( @@ -53,6 +53,11 @@ def _extract_v_corr_idxs(two_view_results) -> dict: return {ij: result.v_corr_idxs for ij, result in two_view_results.items()} +def _identity(x): + """Wrap an eager value as a single delayed node (so it is embedded once, not per consumer).""" + return x + + def _get_image_shapes(loader, image_indices: tuple[int, ...]) -> dict[int, tuple[int, int]]: """Return original (height, width) for each requested image index.""" return {idx: loader.get_image(idx).value_array.shape[:2] for idx in image_indices} @@ -126,6 +131,36 @@ def _scale_camera_intrinsics( raise ValueError(f"Unsupported calibration type: {type(cal)}") +def _predicted_intrinsics_original_res(vggt_result, image_shapes, image_indices) -> dict[int, list[float]]: + """Model-predicted intrinsics rescaled to original image resolution, keyed by global index. + + Offline-replay telemetry: the build overrides the model's predicted intrinsics with the + view-graph/refined focals, so without this capture they are unrecoverable from any export. + """ + global_to_local = {gidx: lidx for lidx, gidx in enumerate(image_indices)} + out: dict[int, list[float]] = {} + for global_idx, camera in vggt_result.cameras.items(): + if global_idx not in image_shapes or global_idx not in global_to_local: + continue + _, orig_W = image_shapes[global_idx] + scaled_W = float(vggt_result.original_coords[global_to_local[global_idx], 4]) + cal = _scale_camera_intrinsics(camera, scale=orig_W / scaled_W).calibration() + out[int(global_idx)] = [cal.fx(), cal.k1(), cal.k2(), cal.px(), cal.py()] + return out + + +def _save_predicted_intrinsics(scene, results_path: Path) -> None: + """Persist the model-predicted intrinsics sidecar (if the build attached one).""" + pred = getattr(scene, "_vggt_predicted_intrinsics", None) + if not pred: + return + import json + + results_path.mkdir(parents=True, exist_ok=True) + with open(results_path / "vggt_predicted_intrinsics.json", "w") as f: + json.dump(pred, f) + + def _build_gtsfm_data_from_vggt_depth( vggt_result: VggtGeometryResult, tracks_2d: list[SfmTrack2d], @@ -182,7 +217,12 @@ def _build_gtsfm_data_from_vggt_depth( points_3d: list[np.ndarray] = [] confidences: list[float] = [] - valid_measurements: list[tuple[int, np.ndarray]] = [] + # Every verified SIFT measurement is a valid BA constraint regardless of whether VGGT + # predicted confident depth at that keypoint; only confident depth anchors the 3D-point + # init. The per-measurement pre-BA reprojection filter then prunes observations whose pose + # is inconsistent — recovering cameras whose keypoints fell in VGGT zero-confidence regions + # (low-texture/transient) but whose pose is sound, while still dropping bad-pose cameras. + all_measurements: list[tuple[int, np.ndarray]] = [] for m in track_2d.measurements: global_idx = m.i @@ -199,16 +239,25 @@ def _build_gtsfm_data_from_vggt_depth( scaled_W, scaled_H = original_coords[local_idx, 4], original_coords[local_idx, 5] u_scale = scaled_W / orig_W if orig_W > 0 else 0.0 v_scale = scaled_H / orig_H if orig_H > 0 else 0.0 - u_c = int(np.clip(round(m.uv[0] * u_scale - left), 0, W_vggt - 1)) - v_c = int(np.clip(round(m.uv[1] * v_scale - top), 0, H_vggt - 1)) + all_measurements.append((global_idx, m.uv)) # original keypoint coords; always a BA constraint + + u_c = int(round(m.uv[0] * u_scale - left)) + v_c = int(round(m.uv[1] * v_scale - top)) + # Skip the depth anchor for keypoints that map OUTSIDE the VGGT dense map (e.g. the cropped-away + # margins under crop / aspect-crop input modes). Clamping to the border pixel (the old behavior) + # would anchor the track's 3D point from an unrelated edge location. The measurement is still + # kept as a BA constraint (appended above) — only the depth anchor is dropped. + if not (0 <= u_c < W_vggt and 0 <= v_c < H_vggt): + continue pt3d = dense_points[local_idx, v_c, u_c] conf = float(depth_confidence[local_idx, v_c, u_c]) if np.isfinite(pt3d).all() and conf > 0.0: points_3d.append(pt3d) confidences.append(conf) - valid_measurements.append((global_idx, m.uv)) # original keypoint coords + # Need at least min_track_length CONFIDENT depths to anchor the 3D point (the total + # measurement count, incl. zero-confidence keypoints, is gated separately below). if len(points_3d) < min_track_length: continue @@ -216,7 +265,7 @@ def _build_gtsfm_data_from_vggt_depth( weights /= weights.sum() point_3d_mean = np.average(points_3d, axis=0, weights=weights) sfm_track = SfmTrack(Point3(*point_3d_mean.astype(float))) - for gidx, uv in valid_measurements: + for gidx, uv in all_measurements: if gidx in cameras: sfm_track.addMeasurement(gidx, Point2(*uv.astype(float))) @@ -224,6 +273,8 @@ def _build_gtsfm_data_from_vggt_depth( gtsfm_data.add_track(sfm_track) logger.info("Built GtsfmData with %d cameras and %d tracks.", len(cameras), gtsfm_data.number_tracks()) + setattr(gtsfm_data, "_vggt_predicted_intrinsics", + _predicted_intrinsics_original_res(vggt_result, image_shapes, image_indices)) return gtsfm_data @@ -259,6 +310,48 @@ def _refine_vggt_intrinsics_via_view_graph( return refined +def _build_gtsfm_data_via_triangulation( + vggt_result: VggtGeometryResult, + tracks_2d: list[SfmTrack2d], + image_shapes: dict[int, tuple[int, int]], + image_indices: tuple[int, ...], + num_images: int, + min_track_length: int = 2, + refined_intrinsics: Optional[dict[int, gtsfm_types.CALIBRATION_TYPE]] = None, +) -> GtsfmData: + """Build BA input from VGGT POSES + SIFT-triangulated structure (not VGGT depth). + + VGGT supplies camera poses (and intrinsics, rescaled to original resolution or replaced by the + view-graph-refined focals); the 3D points are triangulated from the SIFT tracks against those + cameras via `multi_view_retriangulate_from_2d_tracks`. Triangulation's geometric (reprojection) + filtering yields multi-view-consistent structure, replacing the inconsistent VGGT-depth points + that the pre-BA reprojection filter was discarding. + """ + global_to_local = {gidx: lidx for lidx, gidx in enumerate(image_indices)} + + cameras_only = GtsfmData(number_images=num_images) + for global_idx, camera in vggt_result.cameras.items(): + if global_idx in image_shapes and global_idx in global_to_local: + if refined_intrinsics is not None: + camera = type(camera)(camera.pose(), refined_intrinsics[global_idx]) + else: + _, orig_W = image_shapes[global_idx] + scaled_W = float(vggt_result.original_coords[global_to_local[global_idx], 4]) + camera = _scale_camera_intrinsics(camera, scale=orig_W / scaled_W) + cameras_only.add_camera(global_idx, camera) + + result = multi_view_retriangulate_from_2d_tracks(cameras_only, tracks_2d, min_track_length=min_track_length) + setattr(result, "_vggt_predicted_intrinsics", + _predicted_intrinsics_original_res(vggt_result, image_shapes, image_indices)) + logger.info( + "Built GtsfmData via triangulation: %d cameras, %d tracks (from %d input 2D tracks).", + result.number_images(), + result.number_tracks(), + len(tracks_2d), + ) + return result + + class ClusterVGGTWithFrontend(ClusterMVO): """Cluster optimizer that combines a traditional MVO frontend with VGGT poses. @@ -290,7 +383,36 @@ def __init__( pose_angular_error_thresh: float = 3, output_worker: Optional[str] = None, use_view_graph_calibration: bool = False, + # Use a single GLOBAL Fetzer calibration (estimated once over the full verified view graph in + # the SceneOptimizer and supplied via ClusterContext) instead of the per-cluster calibration. + # Per-cluster calibration falls back to VGGT focals for cameras with few in-cluster F-edges; + # the global one never uses VGGT focals (heuristic init, refined over all edges). Requires the + # verified pipeline. Takes precedence over use_view_graph_calibration when set. + use_global_view_graph_calibration: bool = False, use_multi_view_retriangulation: bool = False, + # Build cluster BA structure by triangulating the SIFT tracks against the VGGT poses + # (multi-view-consistent) instead of lifting per-pixel VGGT depth. + use_triangulated_structure: bool = False, + # Reuse the global verified correspondences (computed once in the SceneOptimizer over the full + # verified view graph, supplied via ClusterContext) to build this cluster's 2D tracks, instead of + # re-running the per-cluster correspondence generation + two-view estimation. Per-edge v_corr is + # identical, so it is a pure speedup. Requires the verified pipeline; falls back to the per-cluster + # frontend when the globals are absent. + reuse_global_correspondences: bool = False, + # Run the per-cluster pose+structure BA. OFF keeps raw VGGT poses + global-Fetzer intrinsics + # verbatim (ToL census: the BA degraded 7/8 clusters vs GT by resolving the ~4px pose/Fetzer-K + # inconsistency through the free poses); structure polishing then happens only in the + # pose-pinned merge BAs. + run_per_cluster_ba: bool = True, + # Source of the frozen per-camera focals handed to every cluster (requires + # use_global_view_graph_calibration): + # "fetzer" — global Fetzer self-calibration over the verified graph (legacy). + # "exif" — loader/EXIF intrinsics passed through verbatim. ToL gold audit (592 GT cams, + # exact per-image scales): EXIF median focal err 1.91% (+0.06% bias) vs scipy + # Fetzer 5.26% (−4.0% bias, 74% of cams >3%) — Fetzer DEGRADED the EXIF it + # started from; 56% of verified edges are Fetzer-degenerate (planar/focal- + # unstable). The 1DSfM reference itself used EXIF. + calibration_source: str = "fetzer", ) -> None: super().__init__( correspondence_generator=correspondence_generator, @@ -310,13 +432,23 @@ def __init__( self._input_mode = input_mode self._seed = seed self._use_view_graph_calibration = use_view_graph_calibration + self._use_global_view_graph_calibration = use_global_view_graph_calibration self._use_multi_view_retriangulation = use_multi_view_retriangulation + self._use_triangulated_structure = use_triangulated_structure + self._reuse_global_correspondences = reuse_global_correspondences + self._run_per_cluster_ba = run_per_cluster_ba + if calibration_source not in ("fetzer", "exif"): + raise ValueError(f"calibration_source must be 'fetzer' or 'exif', got {calibration_source!r}") + self._calibration_source = calibration_source self._weights_path = Path(weights_path) if weights_path is not None else None self._loader_kwargs: dict[str, Any] = {} if self._weights_path is not None: self._loader_kwargs["weights_path"] = self._weights_path - model_kwargs = self.geometry_transformer.config.model_ctor_kwargs + # Geometry transformers may not expose a `config` (e.g. VggtOmegaGeometryTransformer); + # treat a missing config as "no extra model ctor kwargs". + _geom_config = getattr(self.geometry_transformer, "config", None) + model_kwargs = _geom_config.model_ctor_kwargs if _geom_config is not None else None if model_kwargs: self._loader_kwargs["model_kwargs"] = model_kwargs @@ -332,11 +464,42 @@ def __repr__(self) -> str: components = [ f"correspondence_generator={self.correspondence_generator}", f"two_view_estimator={self.two_view_estimator}", - f"geometry_transformer={self.geometry_transformer.config}", + # Transformers without a `config` (e.g. VggtOmegaGeometryTransformer) contribute their class + # name to the cache key instead — stable across runs, and distinct from the VGGT config repr. + f"geometry_transformer=" + f"{getattr(self.geometry_transformer, 'config', None) or type(self.geometry_transformer).__name__}", f"ba_options={self.ba_options}", + # Calibration/structure flags change the reconstruction, so include them in the repr that + # seeds the per-cluster cache key (ClusterOptimizerCacher hashes repr(optimizer)). + # Cache-key bump tokens for build changes whose effect isn't otherwise reflected in the repr: + # /allkpts -> VGGT-depth build adds all verified SIFT measurements, not only conf>0 keypoints + # /gcorr -> tracks built from the reused global correspondences (skips per-cluster frontend) + f"calib=(vgc={self._use_view_graph_calibration}," + f"global={self._use_global_view_graph_calibration}," + f"tri={self._use_triangulated_structure}" + f"{'/allkpts' if not self._use_triangulated_structure else ''}" + f"{'/gcorr' if self._reuse_global_correspondences else ''}," + f"mvr={self._use_multi_view_retriangulation}," + f"clusterba={self._run_per_cluster_ba}," + f"src={self._calibration_source})", ] return "ClusterVGGTWithFrontend(\n " + ",\n ".join(components) + "\n)" + @property + def uses_global_view_graph_calibration(self) -> bool: + """Whether this optimizer expects global Fetzer focals from the SceneOptimizer via ClusterContext.""" + return self._use_global_view_graph_calibration + + @property + def calibration_source(self) -> str: + """Source of the frozen global intrinsics: 'fetzer' (self-calibration) or 'exif' (loader passthrough).""" + return self._calibration_source + + @property + def reuses_global_correspondences(self) -> bool: + """Whether this optimizer reuses the SceneOptimizer's global correspondences (via ClusterContext).""" + return self._reuse_global_correspondences + @staticmethod def get_ui_metadata() -> UiMetadata: return UiMetadata( @@ -355,15 +518,44 @@ def create_computation_graph(self, context: ClusterContext) -> ClusterComputatio image_filenames = context.loader.image_filenames() image_names = tuple(str(image_filenames[idx]) for idx in keys) - # Traditional frontend. - frontend_graphs = self._build_frontend_graphs(context) - io_tasks, metrics = self._build_frontend_output_graphs(context, frontend_graphs) + # Build this cluster's 2D tracks. Two paths: + # (A) reuse: subset the SceneOptimizer's global verified correspondences to this cluster's edges + # and build the tracks EAGERLY in the main process — no per-cluster frontend, no scatter. The + # per-edge v_corr is identical to the per-cluster frontend output (same edges, same two-view), + # so this is a pure speedup. (Edges not in the global v_corr -- e.g. bridge edges added after + # verification -- are skipped; with bridge_min_similarity=0 none exist. Verify track parity.) + # (B) per-cluster frontend (correspondence + two-view; cache-hit from the global verification pass). + if self._reuse_global_correspondences and context.global_v_corr_idxs_dict is not None: + cluster_v_corr = { + ij: context.global_v_corr_idxs_dict[ij] + for ij in context.visibility_graph + if ij in context.global_v_corr_idxs_dict + } + tracks_2d = get_2d_tracks(cluster_v_corr, context.global_keypoints) # eager, main process + logger.info( + "♻️ [%s] Reusing global correspondences: %d/%d cluster edges → %d tracks (frontend skipped).", + context.label, + len(cluster_v_corr), + len(context.visibility_graph), + len(tracks_2d), + ) + tracks_2d_graph = delayed(_identity)(tracks_2d) + v_corr_idxs_graph = delayed(_identity)(cluster_v_corr) + padded_keypoints_graph = delayed(_identity)(context.global_keypoints) + io_tasks, metrics = [], [] + else: + frontend_graphs = self._build_frontend_graphs(context) + io_tasks, metrics = self._build_frontend_output_graphs(context, frontend_graphs) + v_corr_idxs_graph = delayed(_extract_v_corr_idxs)(frontend_graphs.two_view_results) + tracks_2d_graph = delayed(get_2d_tracks)(v_corr_idxs_graph, frontend_graphs.padded_keypoints) + padded_keypoints_graph = frontend_graphs.padded_keypoints # VGGT geometry prediction → cameras + dense 3D points. image_batch_graph, original_coords_graph = delayed(_load_vggt_inputs, nout=2)( context.loader, global_indices, mode=self._input_mode, + transformer=self.geometry_transformer, output_root=str(context.output_paths.results), image_names=image_names, ) @@ -379,27 +571,38 @@ def create_computation_graph(self, context: ClusterContext) -> ClusterComputatio cluster_label=context.label, ) - # 3. 2D tracks from frontend correspondences. - v_corr_idxs_graph = delayed(_extract_v_corr_idxs)(frontend_graphs.two_view_results) - tracks_2d_graph = delayed(get_2d_tracks)(v_corr_idxs_graph, frontend_graphs.padded_keypoints) - - # 4. Original image shapes (needed to map frontend pixel coords → VGGT pixel coords). + # Original image shapes (map keypoints → VGGT pixel coords / rescale intrinsics to original res). image_shapes_graph = delayed(_get_image_shapes)(context.loader, global_indices) - # 4b. Optional: refine VGGT's predicted intrinsics via Fetzer joint - # optimization over the frontend's F-matrices (keeps VGGT's predicted poses). + # Intrinsics for the BA cameras (VGGT poses are always kept). Preference order: + # 1. GLOBAL Fetzer focals (estimated once over the full verified graph; never VGGT) if enabled + # and supplied via ClusterContext -- subset to this cluster's cameras. + # 2. PER-CLUSTER Fetzer (refines VGGT focals on this cluster's F-edges; VGGT fallback otherwise). + # 3. None -> the build fn rescales the raw VGGT focal. refined_intrinsics_graph = None - if self._use_view_graph_calibration: + if self._use_global_view_graph_calibration and context.global_refined_intrinsics is not None: + refined_intrinsics_graph = { + idx: context.global_refined_intrinsics[idx] + for idx in global_indices + if idx in context.global_refined_intrinsics + } + elif self._use_view_graph_calibration: refined_intrinsics_graph = delayed(_refine_vggt_intrinsics_via_view_graph)( vggt_result_graph, v_corr_idxs_graph, - frontend_graphs.padded_keypoints, + padded_keypoints_graph, image_shapes_graph, global_indices, ) - # 5. Build GtsfmData: lift 2D tracks to 3D using VGGT depth map. - ba_input_graph = delayed(_build_gtsfm_data_from_vggt_depth)( + # Build BA input. use_triangulated_structure: VGGT poses + SIFT tracks TRIANGULATED against + # those poses (consistent structure). Otherwise: lift the 2D tracks to 3D via the VGGT depth map. + build_ba_input = ( + _build_gtsfm_data_via_triangulation + if self._use_triangulated_structure + else _build_gtsfm_data_from_vggt_depth + ) + ba_input_graph = delayed(build_ba_input)( vggt_result_graph, tracks_2d_graph, image_shapes=image_shapes_graph, @@ -420,6 +623,7 @@ def create_computation_graph(self, context: ClusterContext) -> ClusterComputatio cluster_label=context.label, tracks_2d=tracks_2d_graph, use_multi_view_retriangulation=self._use_multi_view_retriangulation, + run_bundle_adjustment=self._run_per_cluster_ba, ) # 7. Metrics + I/O. @@ -438,6 +642,9 @@ def create_computation_graph(self, context: ClusterContext) -> ClusterComputatio io_tasks.append( delayed(_save_pre_ba_reconstruction_as_text)(pre_ba_result_graph, context.output_paths.results) ) + io_tasks.append( + delayed(_save_predicted_intrinsics)(pre_ba_result_graph, context.output_paths.results) + ) return ClusterComputationGraph( io_tasks=tuple(io_tasks), diff --git a/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism.yaml b/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism.yaml new file mode 100644 index 000000000..98f65b27e --- /dev/null +++ b/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism.yaml @@ -0,0 +1,128 @@ +# VGGT-Omega-with-frontend configuration. +# Poses from VGGT Omega geometry; 2D tracks from the SIFT frontend. +# 3D points are depth-lifted from VGGT Omega dense predictions, then refined by cluster BA. + +# @package _global_ +_target_: gtsfm.scene_optimizer.SceneOptimizer + +loader: + _target_: gtsfm.loader.Olsson + dataset_dir: ??? # Required: set to the dataset root on the command line. + images_dir: null + max_resolution: 760 + +image_pairs_generator: + _target_: gtsfm.retriever.image_pairs_generator.ImagePairsGenerator + global_descriptor: + _target_: gtsfm.frontend.cacher.global_descriptor_cacher.GlobalDescriptorCacher + global_descriptor_obj: + _target_: gtsfm.frontend.global_descriptor.MegaLoc + retriever: + _target_: gtsfm.retriever.Similarity + num_matched: 15 + min_score: 0.3 + batch_size: 16 + +graph_partitioner: + _target_: gtsfm.graph_partitioner.Metis + min_cameras_to_partition: 12 + max_cameras: 40 + split_oversized_nodes: true + +cluster_optimizer: + _target_: gtsfm.cluster_optimizer.Cacher + optimizer: + _target_: gtsfm.cluster_optimizer.VggtOmegaWithFrontend + + correspondence_generator: + _target_: gtsfm.frontend.correspondence_generator.det_desc_correspondence_generator.DetDescCorrespondenceGenerator + detector_descriptor: + _target_: gtsfm.frontend.cacher.detector_descriptor_cacher.DetectorDescriptorCacher + detector_descriptor_obj: + _target_: gtsfm.frontend.detector_descriptor.colmap_sift.ColmapSIFTDetectorDescriptor + max_keypoints: 3000 + + matcher: + _target_: gtsfm.frontend.cacher.matcher_cacher.MatcherCacher + matcher_obj: + _target_: gtsfm.frontend.matcher.twoway_matcher.TwoWayMatcher + ratio_test_threshold: 0.8 + + two_view_estimator: + _target_: gtsfm.two_view_estimator_cacher.TwoViewEstimatorCacher + two_view_estimator_obj: + _target_: gtsfm.two_view_estimator.TwoViewEstimator + bundle_adjust_2view: True + eval_threshold_px: 4 # in px + ba_reproj_error_thresholds: [0.5] + bundle_adjust_2view_maxiters: 100 + + verifier: + _target_: gtsfm.frontend.verifier.ransac.Ransac + use_intrinsics_in_verification: True + estimation_threshold_px: 4 # for H/E/F estimators + + triangulation_options: + _target_: gtsfm.data_association.point3d_initializer.TriangulationOptions + reproj_error_threshold: 100.0 + mode: + _target_: gtsfm.data_association.point3d_initializer.TriangulationSamplingMode + value: NO_RANSAC + + inlier_support_processor: + _target_: gtsfm.two_view_estimator.InlierSupportProcessor + min_num_inliers_est_model: 15 + min_inlier_ratio_est_model: 0.1 + + geometry_transformer: + _target_: gtsfm.frontend.vggt_omega_geometry_transformer.VggtOmegaGeometryTransformer + + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + use_calibration_prior: true + use_pose_prior_all_cameras: true + use_gnc: true + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + + pre_ba_max_reproj_error: 14.0 + post_ba_max_reproj_error: 3.0 + drop_camera_with_no_track: false + min_track_length: 3 + seed: 42 + model_cache_key: null + metric_constructed_only: false + +# --- Merging options --- +merging_options: + _target_: gtsfm.cluster_merging.MergingOptions + run_bundle_adjustment: true + merge_duplicate_tracks: true + drop_child_if_merging_fail: true + drop_outlier_after_camera_merging: true + drop_camera_with_no_track: false + keep_all_cameras: false + plot_reprojection_histograms: true + use_nonlinear_sim3_alignment: true + max_track_correspondences_for_sim3: 150 + scale_and_average_focal_length: false + pre_ba_max_reproj_error: 14.0 + pre_ba_min_track_length: 3 + post_ba_max_reproj_error: 3.0 + min_track_length: 3 + allow_post_ba_reproj_filtering: true + metric_constructed_only: true + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + use_calibration_prior: true + use_pose_prior_all_cameras: true + use_gnc: false + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + +# --- Bridge params --- +bridge_min_similarity: 0.0 +bridge_top_k: 10 +bridge_min_component_size: 3 diff --git a/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_colmapdb.yaml b/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_colmapdb.yaml new file mode 100644 index 000000000..823775c5d --- /dev/null +++ b/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_colmapdb.yaml @@ -0,0 +1,156 @@ +# VGGT-OMEGA + VERIFIED pipeline, fed by an OFFLINE COLMAP database (paper-scale scenes). +# Identical to vggt_omega_sift_frontend_megaloc_phototourism_verified.yaml EXCEPT the frontend: instead of +# GTSFM's Dask SIFT detection/matching/verification (which OOMs on 1000+-image scenes), the verified +# keypoints + matches are read straight from a COLMAP database.db built offline with GPU SIFT + matching. +# - retriever -> ColmapDB: the db's geometrically-verified pairs ARE the view graph. +# - correspondence_generator -> ColmapCorrespondenceGenerator: db keypoints + inlier matches. +# - scene_optimizer detects `produces_verified_correspondences` and SKIPS the Dask two-view pass + the +# all-results gather entirely (reads the db in the main process) — no worker OOM. +# Everything downstream (verified-graph partition, global Fetzer, per-cluster omega, merge, retri) is +# unchanged. db path defaults to /database.db; override the two database_path keys to +# point elsewhere. Build the db with scripts/build_colmap_db.sh. + +# @package _global_ +_target_: gtsfm.scene_optimizer.SceneOptimizer + +loader: + _target_: gtsfm.loader.Olsson + dataset_dir: ??? # Required: set to the dataset root on the command line (must contain database.db + images). + images_dir: null + max_resolution: 518 # VGGT recommended max resolution. + +image_pairs_generator: + _target_: gtsfm.retriever.image_pairs_generator.ImagePairsGenerator + # No global descriptor needed — pairs come from the COLMAP db, not MegaLoc similarity. + global_descriptor: null + retriever: + _target_: gtsfm.retriever.ColmapDB + database_path: ${loader.dataset_dir}/database.db + batch_size: 16 + +graph_partitioner: + _target_: gtsfm.graph_partitioner.Metis + min_cameras_to_partition: 30 + max_cameras: 70 + split_oversized_nodes: true + +cluster_optimizer: + _target_: gtsfm.cluster_optimizer.Cacher + optimizer: + _target_: gtsfm.cluster_optimizer.VggtWithFrontend + + # COLMAP-DB frontend: reads keypoints + ALREADY-VERIFIED inlier matches from the database (rescaled to + # the loader resolution). Marked produces_verified_correspondences=True, so the verified pipeline reads + # it directly and skips the Dask two-view estimation + gather. + correspondence_generator: + _target_: gtsfm.frontend.correspondence_generator.colmap_correspondence_generator.ColmapCorrespondenceGenerator + database_path: ${loader.dataset_dir}/database.db + + # Unused on the db path (the bypass skips two-view estimation; clusters reuse global correspondences), + # but kept so the optimizer instantiates with a valid verifier. + two_view_estimator: + _target_: gtsfm.two_view_estimator_cacher.TwoViewEstimatorCacher + two_view_estimator_obj: + _target_: gtsfm.two_view_estimator.TwoViewEstimator + bundle_adjust_2view: True + eval_threshold_px: 4 + ba_reproj_error_thresholds: [0.5] + bundle_adjust_2view_maxiters: 100 + verifier: + _target_: gtsfm.frontend.verifier.poselib_verifier.PoseLibVerifier + estimation_threshold_px: 2 + triangulation_options: + _target_: gtsfm.data_association.point3d_initializer.TriangulationOptions + reproj_error_threshold: 100.0 + mode: + _target_: gtsfm.data_association.point3d_initializer.TriangulationSamplingMode + value: NO_RANSAC + inlier_support_processor: + _target_: gtsfm.two_view_estimator.InlierSupportProcessor + min_num_inliers_est_model: 30 + min_inlier_ratio_est_model: 0.15 + + # --- VGGT-Omega geometry (peak). Same as the omega verified config. --- + geometry_transformer: + _target_: gtsfm.frontend.vggt_omega_geometry_transformer.VggtOmegaGeometryTransformer + + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + use_calibration_prior: true + calibration_prior_focal_sigma: 5.0 + use_gnc: true + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + + pre_ba_max_reproj_error: 14.0 + post_ba_max_reproj_error: 3.0 + drop_camera_with_no_track: false + min_track_length: 3 + # Golden 1DSfM cluster stage: BA on (EXIF-pinned focals + GNC above), poses free. + run_per_cluster_ba: true + input_mode: crop + seed: 42 + model_cache_key: false # forces the omega transformer to self-load (never the vanilla VGGT loader) + use_view_graph_calibration: true + use_global_view_graph_calibration: true + # EXIF passthrough (ToL gold audit 2026-07-06): EXIF ~1.9% vs Fetzer ~5.3% median focal error + # on 1DSfM. NOTE: the class default is "fetzer" — this key must stay explicit. + calibration_source: exif + use_multi_view_retriangulation: false + # Triangulated structure (NOT depth-lift): the ToL audit measured 87-98% of per-cluster + # structure destroyed by the depth-lift path. Must match the verified config. + use_triangulated_structure: true + # Reuse the global (db) verified correspondences per cluster — required so clusters don't re-run a + # frontend. The colmap-db path makes this the only consumer of the correspondences. + reuse_global_correspondences: true + +# --- Merging options (identical to the omega verified config) --- +merging_options: + _target_: gtsfm.cluster_merging.MergingOptions + run_bundle_adjustment: true + # Post-merge retri BA freedom (defaults preserve current behavior); override per-run, + # e.g. merging_options.retri_free_ba=true for the golden free finale. + retri_free_ba: false + retri_iterations: 1 + merge_duplicate_tracks: true + drop_child_if_merging_fail: true + drop_outlier_after_camera_merging: true + drop_camera_with_no_track: false + keep_all_cameras: false + plot_reprojection_histograms: true + use_nonlinear_sim3_alignment: true + max_track_correspondences_for_sim3: 150 + scale_and_average_focal_length: false + pre_ba_max_reproj_error: 14.0 + pre_ba_min_track_length: 3 + post_ba_max_reproj_error: 3.0 + min_track_length: 3 + allow_post_ba_reproj_filtering: true + metric_constructed_only: true + # OFF per the ToL metrics audit: recovery injected more floaters than it recovered + # (55/79 injected cameras failed to re-triangulate). Matches the verified config. + recover_trackless_cameras_in_retriangulation: false + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + use_calibration_prior: true + calibration_prior_focal_sigma: 10.0 + use_pose_prior_all_cameras: true + use_gnc: false + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + +# --- Bridge params --- +bridge_min_similarity: 0.0 +bridge_top_k: 10 +bridge_min_component_size: 3 + +# --- Verified-viewgraph pipeline --- +use_verified_pipeline: true + +# Tail flags in lockstep with the verified config (2026-07-14 sync). +enable_gid_merge_anchoring: false +run_post_merge_retriangulation: false +enable_boundary_recovery: false +min_triangulation_angle_deg: 1.5 diff --git a/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_colmapdb_megaloc.yaml b/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_colmapdb_megaloc.yaml new file mode 100644 index 000000000..0bd25e907 --- /dev/null +++ b/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_colmapdb_megaloc.yaml @@ -0,0 +1,142 @@ +# VGGT-OMEGA + VERIFIED pipeline, fed by an OFFLINE COLMAP database, with the dense COLMAP view graph +# SPARSIFIED by MegaLoc similarity. Identical to vggt_omega_sift_frontend_megaloc_phototourism_colmapdb.yaml +# EXCEPT the retriever: COLMAP's vocab-tree matching is dense (~140k verified pairs on St Peter's -> a huge +# Metis cluster tree). ColmapDBMegaLoc keeps only the pairs that are BOTH COLMAP-verified AND in MegaLoc's +# per-image top-K (num_matched/min_score) — a strict intersection that sparsifies the view graph while +# still reading correspondences from the db for the surviving pairs. Everything downstream is unchanged. +# db path defaults to /database.db; override the two database_path keys to point +# elsewhere (e.g. a node-local copy). Build the db with scripts/build_colmap_db.sh. + +# @package _global_ +_target_: gtsfm.scene_optimizer.SceneOptimizer + +loader: + _target_: gtsfm.loader.Olsson + dataset_dir: ??? # Required: set to the dataset root on the command line (must contain database.db + images). + images_dir: null + max_resolution: 518 # VGGT recommended max resolution. + +image_pairs_generator: + _target_: gtsfm.retriever.image_pairs_generator.ImagePairsGenerator + # MegaLoc similarity is (re)enabled here purely to SPARSIFY the COLMAP view graph. The hybrid retriever + # keeps each image's top-`num_matched` COLMAP-verified neighbors RANKED by MegaLoc similarity (not a + # strict intersection — that collapsed to a fragmented graph). min_score=0.0 = rank-only (COLMAP already + # gates geometric quality). correspondences still come from the db. + global_descriptor: + _target_: gtsfm.frontend.cacher.global_descriptor_cacher.GlobalDescriptorCacher + global_descriptor_obj: + _target_: gtsfm.frontend.global_descriptor.MegaLoc + retriever: + _target_: gtsfm.retriever.ColmapDBMegaLoc + database_path: ${loader.dataset_dir}/database.db + num_matched: 20 + min_score: 0.0 + batch_size: 16 + +graph_partitioner: + _target_: gtsfm.graph_partitioner.Metis + min_cameras_to_partition: 30 + max_cameras: 70 + split_oversized_nodes: true + +cluster_optimizer: + _target_: gtsfm.cluster_optimizer.Cacher + optimizer: + _target_: gtsfm.cluster_optimizer.VggtWithFrontend + + # COLMAP-DB frontend: reads keypoints + ALREADY-VERIFIED inlier matches from the database (rescaled to + # the loader resolution). Marked produces_verified_correspondences=True, so the verified pipeline reads + # it directly and skips the Dask two-view estimation + gather. + correspondence_generator: + _target_: gtsfm.frontend.correspondence_generator.colmap_correspondence_generator.ColmapCorrespondenceGenerator + database_path: ${loader.dataset_dir}/database.db + + # Unused on the db path (the bypass skips two-view estimation; clusters reuse global correspondences), + # but kept so the optimizer instantiates with a valid verifier. + two_view_estimator: + _target_: gtsfm.two_view_estimator_cacher.TwoViewEstimatorCacher + two_view_estimator_obj: + _target_: gtsfm.two_view_estimator.TwoViewEstimator + bundle_adjust_2view: True + eval_threshold_px: 4 + ba_reproj_error_thresholds: [0.5] + bundle_adjust_2view_maxiters: 100 + verifier: + _target_: gtsfm.frontend.verifier.poselib_verifier.PoseLibVerifier + estimation_threshold_px: 2 + triangulation_options: + _target_: gtsfm.data_association.point3d_initializer.TriangulationOptions + reproj_error_threshold: 100.0 + mode: + _target_: gtsfm.data_association.point3d_initializer.TriangulationSamplingMode + value: NO_RANSAC + inlier_support_processor: + _target_: gtsfm.two_view_estimator.InlierSupportProcessor + min_num_inliers_est_model: 30 + min_inlier_ratio_est_model: 0.15 + + # --- VGGT-Omega geometry (peak). Same as the omega verified config. --- + geometry_transformer: + _target_: gtsfm.frontend.vggt_omega_geometry_transformer.VggtOmegaGeometryTransformer + + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + use_calibration_prior: true + calibration_prior_focal_sigma: 5.0 + use_gnc: true + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + + pre_ba_max_reproj_error: 14.0 + post_ba_max_reproj_error: 3.0 + drop_camera_with_no_track: false + min_track_length: 3 + input_mode: crop + seed: 42 + model_cache_key: false # forces the omega transformer to self-load (never the vanilla VGGT loader) + use_view_graph_calibration: true + use_global_view_graph_calibration: true + use_multi_view_retriangulation: false + use_triangulated_structure: false + # Reuse the global (db) verified correspondences per cluster — required so clusters don't re-run a + # frontend. The colmap-db path makes this the only consumer of the correspondences. + reuse_global_correspondences: true + +# --- Merging options (identical to the omega verified config) --- +merging_options: + _target_: gtsfm.cluster_merging.MergingOptions + run_bundle_adjustment: true + merge_duplicate_tracks: true + drop_child_if_merging_fail: true + drop_outlier_after_camera_merging: true + drop_camera_with_no_track: false + keep_all_cameras: false + plot_reprojection_histograms: true + use_nonlinear_sim3_alignment: true + max_track_correspondences_for_sim3: 150 + scale_and_average_focal_length: false + pre_ba_max_reproj_error: 14.0 + pre_ba_min_track_length: 3 + post_ba_max_reproj_error: 3.0 + min_track_length: 3 + allow_post_ba_reproj_filtering: true + metric_constructed_only: true + recover_trackless_cameras_in_retriangulation: true + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + use_calibration_prior: true + calibration_prior_focal_sigma: 10.0 + use_pose_prior_all_cameras: true + use_gnc: false + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + +# --- Bridge params --- +bridge_min_similarity: 0.0 +bridge_top_k: 10 +bridge_min_component_size: 3 + +# --- Verified-viewgraph pipeline --- +use_verified_pipeline: true diff --git a/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_verified.yaml b/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_verified.yaml new file mode 100644 index 000000000..fc447e37e --- /dev/null +++ b/gtsfm/configs/vggt_omega_sift_frontend_megaloc_phototourism_verified.yaml @@ -0,0 +1,224 @@ +# VGGT-OMEGA-with-frontend configuration (VERIFIED-VIEWGRAPH pipeline; A/B treatment arm). +# Identical to vggt_sift_frontend_megaloc_phototourism_verified.yaml EXCEPT the geometry model is +# VGGT-Omega instead of vanilla VGGT. This is a controlled A/B: the SIFT frontend, verified-graph +# partition, global Fetzer focals, correspondence reuse, and post-merge retriangulation+recovery are +# all unchanged — only the per-cluster pose/depth predictor changes (vggt -> vggt-omega). It runs +# omega geometry through OUR verified optimizer (ClusterVGGTWithFrontend, via the injected +# geometry_transformer), NOT Harneet's ClusterVGGTOmegaWithFrontend (which lacks the verified +# features). Requires the thirdparty/vggt-omega submodule + gated weights (cc-by-nc-4.0); see +# scripts/download_model_weights.sh --hf_token . CUDA-only. + +# @package _global_ +_target_: gtsfm.scene_optimizer.SceneOptimizer + +loader: + _target_: gtsfm.loader.Olsson + dataset_dir: ??? # Required: set to the dataset root on the command line. + images_dir: null + max_resolution: 518 # VGGT recommended max resolution. + +image_pairs_generator: + _target_: gtsfm.retriever.image_pairs_generator.ImagePairsGenerator + global_descriptor: + _target_: gtsfm.frontend.cacher.global_descriptor_cacher.GlobalDescriptorCacher + global_descriptor_obj: + _target_: gtsfm.frontend.global_descriptor.MegaLoc + retriever: + _target_: gtsfm.retriever.Similarity + num_matched: 120 # K is the lever (validated live 2026-07-05: the mis-seated block at K=60 seats at K=120 + min_score: 0.15 # — its bridge edges rank 60-300; ~52k pairs vs 114k at 200/0.10, same structural win) + batch_size: 16 + +graph_partitioner: + _target_: gtsfm.graph_partitioner.Metis + min_cameras_to_partition: 30 + max_cameras: 70 + split_oversized_nodes: true + +cluster_optimizer: + _target_: gtsfm.cluster_optimizer.Cacher + optimizer: + _target_: gtsfm.cluster_optimizer.VggtWithFrontend + + correspondence_generator: + _target_: gtsfm.frontend.correspondence_generator.det_desc_correspondence_generator.DetDescCorrespondenceGenerator + detector_descriptor: + _target_: gtsfm.frontend.cacher.detector_descriptor_cacher.DetectorDescriptorCacher + detector_descriptor_obj: + _target_: gtsfm.frontend.detector_descriptor.colmap_sift.ColmapSIFTDetectorDescriptor + max_keypoints: 8192 + + matcher: + _target_: gtsfm.frontend.cacher.matcher_cacher.MatcherCacher + matcher_obj: + _target_: gtsfm.frontend.matcher.twoway_matcher.TwoWayMatcher + ratio_test_threshold: 0.8 + + two_view_estimator: + _target_: gtsfm.two_view_estimator_cacher.TwoViewEstimatorCacher + two_view_estimator_obj: + _target_: gtsfm.two_view_estimator.TwoViewEstimator + bundle_adjust_2view: True + eval_threshold_px: 4 # in px + ba_reproj_error_thresholds: [0.5] + bundle_adjust_2view_maxiters: 100 + + verifier: + # PoseLib 5-point + LO-RANSAC + 2-view bundle (gp-glomap-parity peak verifier). + _target_: gtsfm.frontend.verifier.poselib_verifier.PoseLibVerifier + estimation_threshold_px: 2 + + triangulation_options: + _target_: gtsfm.data_association.point3d_initializer.TriangulationOptions + reproj_error_threshold: 100.0 + mode: + _target_: gtsfm.data_association.point3d_initializer.TriangulationSamplingMode + value: NO_RANSAC + + inlier_support_processor: + _target_: gtsfm.two_view_estimator.InlierSupportProcessor + min_num_inliers_est_model: 30 + min_inlier_ratio_est_model: 0.15 + + # --- THE ONLY GEOMETRY CHANGE vs the VGGT verified config: swap the predictor to VGGT-Omega. --- + # VggtOmegaGeometryTransformer has no `config` block (the optimizer guards the missing attr) and + # self-loads its model when the optimizer passes model=None (see model_cache_key below). + geometry_transformer: + _target_: gtsfm.frontend.vggt_omega_geometry_transformer.VggtOmegaGeometryTransformer + + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + # Anchor each focal at its (global-)Fetzer value so the per-cluster BA optimizes POSES, not focals. + # Without this, each cluster's BA drifts the same camera's focal differently (focal/depth ambiguity), + # so parent and child versions of a separator camera diverge -> noisier Sim3 merge. Pinned tighter + # (5px) than the merge BA (10px) since focals should stay at Fetzer upstream. + use_calibration_prior: true + calibration_prior_focal_sigma: 5.0 + use_gnc: true + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + + pre_ba_max_reproj_error: 14.0 + post_ba_max_reproj_error: 3.0 + drop_camera_with_no_track: false + min_track_length: 3 + # PER-CLUSTER BA back ON (2026-07-06 live A/B): the BA-off arm regressed — lost bridge, new + # floaters, flipped cameras. The census (BA degraded 7/8 clusters' survivor accuracy) was real + # but had SURVIVOR BIAS: cluster BA + its 3px post-filter is also the pipeline's main early + # CULLING stage (flipped/garbage VGGT cameras lose their tracks there and die; without it they + # ride to the final model as floaters/flips, and merge BAs exclude rather than fix low-track + # cams). Keep BA for its culling; the accuracy damage it caused shrinks with EXIF K (the ~4px + # inconsistency it mis-attributed to poses was largely Fetzer focal error). BA-off remains a + # research direction pending a replacement culling stage (post-cluster consistency filter). + run_per_cluster_ba: true + input_mode: crop + seed: 42 + # false => our optimizer's VGGT loader is bypassed and predict() is called with model=None, so the + # OMEGA transformer self-loads (and worker-caches) its own model. With null it would build a + # ("default_vggt_loader", ...) key and load VANILLA VGGT — silently the wrong model. Must be false. + model_cache_key: false + # Refine VGGT predicted intrinsics via scipy Fetzer focal optimization on the verified F-matrices. + use_view_graph_calibration: true + # Estimate focals with a single GLOBAL Fetzer over the full verified view graph (never falls back to + # VGGT focals like the per-cluster path does). Takes precedence over use_view_graph_calibration. + use_global_view_graph_calibration: true + # EXIF PASSTHROUGH (ToL gold audit, 2026-07-06): loader/EXIF focals are 1.91% median error vs + # 1DSfM GT (unbiased), while the scipy Fetzer values are 5.26% (−4% systematic bias; 56% of + # verified edges are Fetzer-degenerate and poison the ungated joint solve). The 1DSfM reference + # used EXIF too. Frozen-K coherence is unchanged — clusters get bit-identical EXIF K per camera. + # No-EXIF cams keep the loader default (large error, small population); gated-Fetzer Class-B + # rescue is post-deadline work. Flip to "fetzer" to A/B the old behavior. + calibration_source: exif + # Re-triangulate union-find 2D tracks against post-BA cameras and run another BA. + use_multi_view_retriangulation: false + # Per-cluster 3D init: true = VGGT(-Omega) POSES + SIFT tracks TRIANGULATED against them (geometric, + # multi-view-consistent structure). The depth-lift path (false) married each verified track to a single + # monocular depth guess; when that guess reprojected >14px the pre-BA filter deleted the point AND the + # verified constraint with it — the ToL audit measured 87-98% of per-cluster structure destroyed this + # way (26 tracks/cam at merge nodes vs >=51 healthy; 41/117 Sim3 merges left with 0 correspondences). + # Triangulated points pass the reproj filters by construction, so the verified viewgraph's constraints + # actually reach the merges. Same cost as depth-lift (one BA per cluster; triangulation is seconds). + use_triangulated_structure: true + # Reuse the global verified correspondences (computed once over the full verified view graph) to build + # each cluster's 2D tracks, instead of re-running the per-cluster correspondence generation + two-view + # estimation. Pure speedup: per-edge v_corr is identical. Requires use_verified_pipeline. + reuse_global_correspondences: true + +# --- Merging options --- +merging_options: + _target_: gtsfm.cluster_merging.MergingOptions + run_bundle_adjustment: true + # Post-merge retri BA freedom (defaults preserve current behavior). retri_free_ba drops the + # calib/pose priors in the FINAL full-scene solve (GLOMAP-style; BM replay: AUC@10 0.48->0.79); + # retri_iterations alternates retriangulation with that BA. Override per-run for IMC campaigns. + retri_free_ba: false + retri_iterations: 1 + merge_duplicate_tracks: true + drop_child_if_merging_fail: true + drop_outlier_after_camera_merging: true + drop_camera_with_no_track: false + keep_all_cameras: false + plot_reprojection_histograms: true + use_nonlinear_sim3_alignment: true + max_track_correspondences_for_sim3: 150 + scale_and_average_focal_length: false + pre_ba_max_reproj_error: 14.0 + pre_ba_min_track_length: 3 + post_ba_max_reproj_error: 3.0 + min_track_length: 3 + allow_post_ba_reproj_filtering: true + metric_constructed_only: true + # Sim3 scale band REMOVED (defaults to [0, inf); Roman Forum audit 2026-07-07): VGGT cluster + # scale is arbitrary per batch, so the old [0.25, 4] band executed lawful seats (~1,200 cams). + # Diverged seats are caught by the structureless guards + merge BA + reproj filters instead. + # Trackless-camera recovery is OFF: the Tower of London metrics audit showed it injects more damage than + # it recovers — 55 of 79 injected cameras failed to re-triangulate (0 retri tracks despite touching + # 2000-4800 global tracks each => their merged poses were wrong) and their bad poses leaked into the + # output as floaters; only 24/79 recovered. With a healthy retrieval/partition (num_matched=60, + # max_cameras=150) cameras keep their tracks, so there is little left to recover. Flip back to true only + # if a scene shows a large genuinely-trackless camera drop. + recover_trackless_cameras_in_retriangulation: false + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + # Anchor focals at their (good) per-cluster-BA / Fetzer values so the merge + post-merge-retri BA + # optimize POSES, not focals. Without this the free-focal BA distorts good focals (0.9-2.4% -> up to + # 100% err vs GLOMAP) to absorb Sim3 pose error, inflating reproj and dropping cameras at the 3px filter. + use_calibration_prior: true + # Back to 10 (2026-07-06): σ=2 was part of the regressed BA-off arm (less room for the pinned + # merge BA to polish -> structure loss at the 3px filters -> bridge child starved+dropped). + # σ=10 is the proven golden-run value. Revisit tightening only with a merge-survival guard. + calibration_prior_focal_sigma: 10.0 + use_pose_prior_all_cameras: true + use_gnc: false + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + +# --- Bridge params --- +bridge_min_similarity: 0.0 +bridge_top_k: 10 +bridge_min_component_size: 3 + +# --- Verified-viewgraph pipeline (A/B treatment): verified-graph partition + post-merge retriangulation+BA --- +use_verified_pipeline: true + +# Gid anchoring RE-ENABLED (2026-07): _track_gid now MAJORITY-VOTES a track's identity over ALL its +# measurements instead of returning the first index hit, so a single pixel-bin collision can no longer +# mint phantom parent<->child correspondences (offline replay saw zero-overlap children arrive with +# "150 matches"). With honest anchor counts the MERGE_GUARDs act on real evidence; offline replay of +# the instrumented ToL drop: strays 54->30, aligned inliers 230->289. +enable_gid_merge_anchoring: false # peak-structure legacy merge for this A/B (majority-vote fix stays in code) +# Retri stays OFF; the boundary-recovery stage below is the post-merge structure/camera recovery path. +run_post_merge_retriangulation: false +# Post-root-merge boundary recovery (offline-validated exp5 recipe): DLT-triangulate boundary global +# tracks (>=3 posed views, <3px mean reproj) against the merged cameras, RANSAC-DLT resect unposed +# cameras (min 10 inliers @4px) against the cluster-3D ∪ fresh-3D UNION (an island camera had 508 +# anchors in cluster-3D vs 95 fresh-only — the union is what makes resection work), iterate, then one +# BA with the merge ba_options. Requires enable_gid_merge_anchoring (reads merged 3D by global id). +enable_boundary_recovery: false # off for this experiment; re-enable after retrieval A/B +# Export an angle-filtered copy of the final merged scene (merged_anglefiltered/) dropping tracks +# whose max pairwise triangulation angle is under this (COLMAP's default cutoff). Validated on the +# 120/0.15 ToL drop: killed 16% of tracks — almost entirely the low-parallax fuzz/spray that passes +# the 3px reproj filter with unconstrained depth. Output-side only; merged/ is still written unfiltered. +min_triangulation_angle_deg: 1.5 diff --git a/gtsfm/configs/vggt_sift_frontend_megaloc_phototourism.yaml b/gtsfm/configs/vggt_sift_frontend_megaloc_phototourism.yaml index 706030fdb..1fd741cae 100644 --- a/gtsfm/configs/vggt_sift_frontend_megaloc_phototourism.yaml +++ b/gtsfm/configs/vggt_sift_frontend_megaloc_phototourism.yaml @@ -19,14 +19,14 @@ image_pairs_generator: _target_: gtsfm.frontend.global_descriptor.MegaLoc retriever: _target_: gtsfm.retriever.Similarity - num_matched: 15 - min_score: 0.5 + num_matched: 100 + min_score: 0.15 batch_size: 16 graph_partitioner: _target_: gtsfm.graph_partitioner.Metis - min_cameras_to_partition: 12 - max_cameras: 40 + min_cameras_to_partition: 30 + max_cameras: 70 split_oversized_nodes: true cluster_optimizer: @@ -39,8 +39,8 @@ cluster_optimizer: detector_descriptor: _target_: gtsfm.frontend.cacher.detector_descriptor_cacher.DetectorDescriptorCacher detector_descriptor_obj: - _target_: gtsfm.frontend.detector_descriptor.sift.SIFTDetectorDescriptor - max_keypoints: 5000 + _target_: gtsfm.frontend.detector_descriptor.colmap_sift.ColmapSIFTDetectorDescriptor + max_keypoints: 8192 matcher: _target_: gtsfm.frontend.cacher.matcher_cacher.MatcherCacher @@ -58,9 +58,9 @@ cluster_optimizer: bundle_adjust_2view_maxiters: 100 verifier: - _target_: gtsfm.frontend.verifier.ransac.Ransac - use_intrinsics_in_verification: True - estimation_threshold_px: 4 # for H/E/F estimators + # PoseLib 5-point + LO-RANSAC + 2-view bundle (gp-glomap-parity peak verifier). + _target_: gtsfm.frontend.verifier.poselib_verifier.PoseLibVerifier + estimation_threshold_px: 2 triangulation_options: _target_: gtsfm.data_association.point3d_initializer.TriangulationOptions @@ -71,8 +71,8 @@ cluster_optimizer: inlier_support_processor: _target_: gtsfm.two_view_estimator.InlierSupportProcessor - min_num_inliers_est_model: 15 - min_inlier_ratio_est_model: 0.1 + min_num_inliers_est_model: 30 + min_inlier_ratio_est_model: 0.15 geometry_transformer: _target_: gtsfm.frontend.vggt_geometry_transformer.VggtGeometryTransformer @@ -85,7 +85,14 @@ cluster_optimizer: ba_options: _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions shared_calib: false - use_calibration_prior: false + # Anchor each focal at its (per-cluster-)Fetzer value so the per-cluster BA optimizes POSES, not + # focals. Without this, each cluster's BA drifts the same camera's focal differently (focal/depth + # ambiguity), so parent and child versions of a separator camera diverge -> noisier Sim3 merge. + # Pinned tighter (5px) than the merge BA (10px) since focals should stay at Fetzer upstream. + # NOTE: no global Fetzer here, so cameras with few F-edges anchor at the per-cluster VGGT fallback; + # the soft prior keeps that low-risk. + use_calibration_prior: true + calibration_prior_focal_sigma: 5.0 use_gnc: true gnc_loss: TLS factor_weight_outlier_threshold: 1e-6 @@ -97,10 +104,13 @@ cluster_optimizer: input_mode: crop seed: 42 model_cache_key: null - # Refine VGGT predicted intrinsics via Fetzer joint optimization on F-matrices. - use_view_graph_calibration: false + # Refine VGGT predicted intrinsics via scipy Fetzer focal optimization on the verified F-matrices. + use_view_graph_calibration: true # Re-triangulate union-find 2D tracks against post-BA cameras and run another BA. use_multi_view_retriangulation: false + # Build cluster BA structure by triangulating SIFT tracks against VGGT poses (multi-view + # consistent) instead of lifting per-pixel VGGT depth — fixes pre-BA reproj track loss. + use_triangulated_structure: true # --- Merging options --- merging_options: @@ -124,8 +134,11 @@ merging_options: ba_options: _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions shared_calib: false - use_calibration_prior: false - use_pose_prior: true + # Anchor focals at their (good) per-cluster-BA / Fetzer values so the merge + post-merge-retri BA + # optimize POSES, not focals (prevents free-focal BA from distorting good focals to absorb Sim3 pose error). + use_calibration_prior: true + calibration_prior_focal_sigma: 10.0 + use_pose_prior_all_cameras: true use_gnc: false gnc_loss: TLS factor_weight_outlier_threshold: 1e-6 diff --git a/gtsfm/configs/vggt_sift_frontend_megaloc_phototourism_verified.yaml b/gtsfm/configs/vggt_sift_frontend_megaloc_phototourism_verified.yaml new file mode 100644 index 000000000..373e62b2d --- /dev/null +++ b/gtsfm/configs/vggt_sift_frontend_megaloc_phototourism_verified.yaml @@ -0,0 +1,170 @@ +# VGGT-with-frontend configuration (VERIFIED-VIEWGRAPH pipeline; A/B treatment arm). +# Identical to vggt_sift_frontend_megaloc_phototourism.yaml, plus use_verified_pipeline: true, which: +# (a) runs a global two-view verification pass over the full MegaLoc retrieval graph and partitions +# METIS on the VERIFIED subgraph (edges where TwoViewResult.valid()), and +# (b) retriangulates the SIFT-verified multiview tracks against the merged VGGT poses + BA after the +# hierarchical merge (structure refinement), written to results/merged_retriangulated/. +# Baseline arm = vggt_sift_frontend_megaloc_phototourism.yaml (flag defaults to false). + +# @package _global_ +_target_: gtsfm.scene_optimizer.SceneOptimizer + +loader: + _target_: gtsfm.loader.Olsson + dataset_dir: ??? # Required: set to the dataset root on the command line. + images_dir: null + max_resolution: 518 # VGGT recommended max resolution. + +image_pairs_generator: + _target_: gtsfm.retriever.image_pairs_generator.ImagePairsGenerator + global_descriptor: + _target_: gtsfm.frontend.cacher.global_descriptor_cacher.GlobalDescriptorCacher + global_descriptor_obj: + _target_: gtsfm.frontend.global_descriptor.MegaLoc + retriever: + _target_: gtsfm.retriever.Similarity + num_matched: 100 + min_score: 0.15 + batch_size: 16 + +graph_partitioner: + _target_: gtsfm.graph_partitioner.Metis + min_cameras_to_partition: 30 + max_cameras: 70 + split_oversized_nodes: true + +cluster_optimizer: + _target_: gtsfm.cluster_optimizer.Cacher + optimizer: + _target_: gtsfm.cluster_optimizer.VggtWithFrontend + + correspondence_generator: + _target_: gtsfm.frontend.correspondence_generator.det_desc_correspondence_generator.DetDescCorrespondenceGenerator + detector_descriptor: + _target_: gtsfm.frontend.cacher.detector_descriptor_cacher.DetectorDescriptorCacher + detector_descriptor_obj: + _target_: gtsfm.frontend.detector_descriptor.colmap_sift.ColmapSIFTDetectorDescriptor + max_keypoints: 8192 + + matcher: + _target_: gtsfm.frontend.cacher.matcher_cacher.MatcherCacher + matcher_obj: + _target_: gtsfm.frontend.matcher.twoway_matcher.TwoWayMatcher + ratio_test_threshold: 0.8 + + two_view_estimator: + _target_: gtsfm.two_view_estimator_cacher.TwoViewEstimatorCacher + two_view_estimator_obj: + _target_: gtsfm.two_view_estimator.TwoViewEstimator + bundle_adjust_2view: True + eval_threshold_px: 4 # in px + ba_reproj_error_thresholds: [0.5] + bundle_adjust_2view_maxiters: 100 + + verifier: + # PoseLib 5-point + LO-RANSAC + 2-view bundle (gp-glomap-parity peak verifier). + _target_: gtsfm.frontend.verifier.poselib_verifier.PoseLibVerifier + estimation_threshold_px: 2 + + triangulation_options: + _target_: gtsfm.data_association.point3d_initializer.TriangulationOptions + reproj_error_threshold: 100.0 + mode: + _target_: gtsfm.data_association.point3d_initializer.TriangulationSamplingMode + value: NO_RANSAC + + inlier_support_processor: + _target_: gtsfm.two_view_estimator.InlierSupportProcessor + min_num_inliers_est_model: 30 + min_inlier_ratio_est_model: 0.15 + + geometry_transformer: + _target_: gtsfm.frontend.vggt_geometry_transformer.VggtGeometryTransformer + config: + _target_: gtsfm.frontend.vggt_geometry_transformer.VggtGeometryConfig + confidence_threshold: 0.1 + max_num_points: 100000 + seed: 42 + + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + # Anchor each focal at its (global-)Fetzer value so the per-cluster BA optimizes POSES, not focals. + # Without this, each cluster's BA drifts the same camera's focal differently (focal/depth ambiguity), + # so parent and child versions of a separator camera diverge -> noisier Sim3 merge. Pinned tighter + # (5px) than the merge BA (10px) since focals should stay at Fetzer upstream. + use_calibration_prior: true + calibration_prior_focal_sigma: 5.0 + use_gnc: true + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + + pre_ba_max_reproj_error: 14.0 + post_ba_max_reproj_error: 3.0 + drop_camera_with_no_track: false + min_track_length: 3 + input_mode: crop + seed: 42 + model_cache_key: null + # Refine VGGT predicted intrinsics via scipy Fetzer focal optimization on the verified F-matrices. + use_view_graph_calibration: true + # Estimate focals with a single GLOBAL Fetzer over the full verified view graph (never falls back to + # VGGT focals like the per-cluster path does). Takes precedence over use_view_graph_calibration. + use_global_view_graph_calibration: true + # Re-triangulate union-find 2D tracks against post-BA cameras and run another BA. + use_multi_view_retriangulation: false + # Per-cluster 3D init: false = lift per-pixel VGGT predicted depth (Akshay's original, the known-good + # 219-cam / AUC@3 0.675 baseline). The triangulated-structure path (true) inflated clusters to 11k-15k + # tracks → 85-200s BAs → worker OOM / dask FutureCancelledError, with no proven gain over 219. Reverted + # to VGGT depth; the focal-flow fix (global Fetzer + calibration priors) still applies on this path. + use_triangulated_structure: false + # Reuse the global verified correspondences (computed once over the full verified view graph) to build + # each cluster's 2D tracks, instead of re-running the per-cluster correspondence generation + two-view + # estimation (which only cache-reads the same data, serially + redundantly across overlapping clusters). + # Pure speedup: per-edge v_corr is identical. Requires use_verified_pipeline. + reuse_global_correspondences: true + +# --- Merging options --- +merging_options: + _target_: gtsfm.cluster_merging.MergingOptions + run_bundle_adjustment: true + merge_duplicate_tracks: true + drop_child_if_merging_fail: true + drop_outlier_after_camera_merging: true + drop_camera_with_no_track: false + keep_all_cameras: false + plot_reprojection_histograms: true + use_nonlinear_sim3_alignment: true + max_track_correspondences_for_sim3: 150 + scale_and_average_focal_length: false + pre_ba_max_reproj_error: 14.0 + pre_ba_min_track_length: 3 + post_ba_max_reproj_error: 3.0 + min_track_length: 3 + allow_post_ba_reproj_filtering: true + metric_constructed_only: true + # Recover good-pose / no-VGGT-depth cameras (left track-less at construction, e.g. Brussels 104/207) + # by injecting their merged poses into the post-merge retriangulation, which geometrically re-triangulates + # their global 2D tracks (depth-confidence independent). Cameras that still fail to triangulate are + # cleanly dropped. Orthogonal to the per-cluster depth gate; the constructed 229 are untouched. + recover_trackless_cameras_in_retriangulation: true + ba_options: + _target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptions + shared_calib: false + # Anchor focals at their (good) per-cluster-BA / Fetzer values so the merge + post-merge-retri BA + # optimize POSES, not focals. Without this the free-focal BA distorts good focals (0.9-2.4% -> up to + # 100% err vs GLOMAP) to absorb Sim3 pose error, inflating reproj and dropping cameras at the 3px filter. + use_calibration_prior: true + calibration_prior_focal_sigma: 10.0 + use_pose_prior_all_cameras: true + use_gnc: false + gnc_loss: TLS + factor_weight_outlier_threshold: 1e-6 + +# --- Bridge params --- +bridge_min_similarity: 0.0 +bridge_top_k: 10 +bridge_min_component_size: 3 + +# --- Verified-viewgraph pipeline (A/B treatment): verified-graph partition + post-merge retriangulation+BA --- +use_verified_pipeline: true diff --git a/gtsfm/frontend/cacher/detector_descriptor_cacher.py b/gtsfm/frontend/cacher/detector_descriptor_cacher.py index 2bf9df339..bffbf6983 100644 --- a/gtsfm/frontend/cacher/detector_descriptor_cacher.py +++ b/gtsfm/frontend/cacher/detector_descriptor_cacher.py @@ -22,7 +22,7 @@ logger = logger_utils.get_logger() -CACHE_ROOT_PATH = Path(__file__).resolve().parent.parent.parent.parent / "cache" +CACHE_ROOT_PATH = cache_utils.get_cache_root() # honors GTSFM_CACHE_ROOT env var class DetectorDescriptorCacher(DetectorDescriptorBase): diff --git a/gtsfm/frontend/cacher/global_descriptor_cacher.py b/gtsfm/frontend/cacher/global_descriptor_cacher.py index dc2be4e3b..f54590b72 100644 --- a/gtsfm/frontend/cacher/global_descriptor_cacher.py +++ b/gtsfm/frontend/cacher/global_descriptor_cacher.py @@ -24,7 +24,7 @@ logger = logger_utils.get_logger() -CACHE_ROOT_PATH = Path(__file__).resolve().parent.parent.parent.parent / "cache" +CACHE_ROOT_PATH = cache_utils.get_cache_root() # honors GTSFM_CACHE_ROOT env var class GlobalDescriptorCacher(GlobalDescriptorBase): diff --git a/gtsfm/frontend/cacher/image_matcher_cacher.py b/gtsfm/frontend/cacher/image_matcher_cacher.py index 2d9e6b1a4..56b203b8b 100644 --- a/gtsfm/frontend/cacher/image_matcher_cacher.py +++ b/gtsfm/frontend/cacher/image_matcher_cacher.py @@ -17,7 +17,7 @@ logger = logger_utils.get_logger() -CACHE_ROOT_PATH = Path(__file__).resolve().parent.parent.parent.parent / "cache" +CACHE_ROOT_PATH = cache_utils.get_cache_root() # honors GTSFM_CACHE_ROOT env var class ImageMatcherCacher(ImageMatcherBase): diff --git a/gtsfm/frontend/cacher/matcher_cacher.py b/gtsfm/frontend/cacher/matcher_cacher.py index 8bfba5808..0fae92ded 100644 --- a/gtsfm/frontend/cacher/matcher_cacher.py +++ b/gtsfm/frontend/cacher/matcher_cacher.py @@ -20,7 +20,7 @@ logger = logger_utils.get_logger() -CACHE_ROOT_PATH = Path(__file__).resolve().parent.parent.parent.parent / "cache" +CACHE_ROOT_PATH = cache_utils.get_cache_root() # honors GTSFM_CACHE_ROOT env var NUM_KEYPOINTS_TO_SAMPLE_FOR_HASH = 10 diff --git a/gtsfm/frontend/correspondence_generator/colmap_correspondence_generator.py b/gtsfm/frontend/correspondence_generator/colmap_correspondence_generator.py index 8a71be5ec..06116da6e 100644 --- a/gtsfm/frontend/correspondence_generator/colmap_correspondence_generator.py +++ b/gtsfm/frontend/correspondence_generator/colmap_correspondence_generator.py @@ -28,15 +28,24 @@ class ColmapCorrespondenceGenerator(CorrespondenceGeneratorBase): """Load correspondences from Colmap DB.""" + # The matches read from `two_view_geometries` are already geometrically verified, so the verified + # pipeline can use them directly and skip its Dask two-view estimation + gather. + produces_verified_correspondences: bool = True + def __init__(self, database_path: str) -> None: """Initialize the correspondence generator with the Colmap DB. Args: database_path: path of the Colmap DB. """ - self._pycolmap_db = pycolmap.Database(database_path) + self._database_path = database_path + self._open_db() + + def _open_db(self) -> None: + """Open the pycolmap db handle and load all keypoints (on construction and after unpickle).""" + self._pycolmap_db = pycolmap.Database(self._database_path) # Note(Ayush): using SQLite3 to load keypoints because PyColmap does not expose bindings. - raw_db = sqlite3.connect(database_path) + raw_db = sqlite3.connect(self._database_path) self._keypoints_dict: Dict[int, np.ndarray] = { image_id: np.frombuffer(data, dtype=np.float32).reshape(rows, -1) for image_id, rows, data in raw_db.execute("SELECT image_id, rows, data FROM keypoints") @@ -50,6 +59,21 @@ def __init__(self, database_path: str) -> None: self._pycolmap_db.num_verified_image_pairs, ) + def __getstate__(self) -> dict: + """Drop the unpicklable pycolmap.Database (and the large, re-derivable keypoints cache) so this + generator can be embedded in a Dask task graph. The colmap-db frontend runs in the MAIN process + (reuse_global_correspondences), so a worker copy never touches the db; _ensure_db re-opens it + lazily if a method is ever actually called on a worker.""" + state = self.__dict__.copy() + state["_pycolmap_db"] = None + state["_keypoints_dict"] = None + return state + + def _ensure_db(self) -> None: + """Re-open the db if this instance was unpickled (e.g. shipped to a Dask worker) with no handle.""" + if getattr(self, "_pycolmap_db", None) is None: + self._open_db() + def _read_keypoints(self, image: Image) -> Keypoints: """ Read keypoints from a pycolmap.Image object. @@ -137,6 +161,7 @@ def generate_correspondences( List of keypoints, one entry for each input images. Putative correspondence as indices of keypoints, for pairs of images. """ + self._ensure_db() # re-open the db if this generator was unpickled onto a worker # Note: we will end up reading verified correspondences from the colmap DB. images_actual = client.gather(images) diff --git a/gtsfm/frontend/correspondence_generator/correspondence_generator_base.py b/gtsfm/frontend/correspondence_generator/correspondence_generator_base.py index 5ff7dbbb0..a1baf16a1 100644 --- a/gtsfm/frontend/correspondence_generator/correspondence_generator_base.py +++ b/gtsfm/frontend/correspondence_generator/correspondence_generator_base.py @@ -9,6 +9,7 @@ import numpy as np from dask.distributed import Client, Future +from gtsfm.common.image import Image from gtsfm.common.keypoints import Keypoints from gtsfm.products.visibility_graph import VisibilityGraph @@ -16,6 +17,12 @@ class CorrespondenceGeneratorBase: """Base class for correspondence generators.""" + # Whether generate_correspondences() returns ALREADY-VERIFIED correspondences (e.g. read from a COLMAP + # database's two_view_geometries). When True, the verified pipeline reads them directly in the main + # process and skips its Dask two-view estimation + the all-results gather (which OOMs at scale). + # Default False: correspondences are putative and must be geometrically verified downstream. + produces_verified_correspondences: bool = False + @abstractmethod def generate_correspondences( self, @@ -34,3 +41,22 @@ def generate_correspondences( List of keypoints, one entry for each input images. Putative correspondence as indices of keypoints, for pairs of images. """ + + def generate_correspondences_inline( + self, + images: List[Image], + visibility_graph: VisibilityGraph, + ) -> Tuple[List[Keypoints], Dict[Tuple[int, int], np.ndarray]]: + """Inline (no-Dask) correspondence generation over already-materialized images. + + Same contract as ``generate_correspondences`` but takes concrete ``Image`` objects (indexed by + position) and runs detection/matching in plain loops — no ``client.submit``/``gather``. This is the + cluster-frontend path; it avoids the ``worker_client()`` tasks-launching-tasks pattern that holds + every feature/correspondence future resident on one worker during a nested gather. + + Subclasses used by the cluster pipeline must override this. The default raises so an unsupported + generator fails loudly rather than silently falling back to the fragile path. + """ + raise NotImplementedError( + f"{type(self).__name__} does not implement generate_correspondences_inline()." + ) diff --git a/gtsfm/frontend/correspondence_generator/det_desc_correspondence_generator.py b/gtsfm/frontend/correspondence_generator/det_desc_correspondence_generator.py index d971c71c8..f555c1f26 100644 --- a/gtsfm/frontend/correspondence_generator/det_desc_correspondence_generator.py +++ b/gtsfm/frontend/correspondence_generator/det_desc_correspondence_generator.py @@ -3,11 +3,13 @@ Authors: John Lambert """ +import time from typing import Dict, List, Tuple import numpy as np from dask.distributed import Client, Future +import gtsfm.utils.logger as logger_utils from gtsfm.common.image import Image from gtsfm.common.keypoints import Keypoints from gtsfm.frontend.correspondence_generator.correspondence_generator_base import CorrespondenceGeneratorBase @@ -15,6 +17,8 @@ from gtsfm.frontend.matcher.matcher_base import MatcherBase from gtsfm.products.visibility_graph import VisibilityGraph +logger = logger_utils.get_logger() + class DetDescCorrespondenceGenerator(CorrespondenceGeneratorBase): """Traditional pair-wise matching of descriptors.""" @@ -85,3 +89,59 @@ def apply_matcher( keypoints_list = client.gather(keypoints_futures) return keypoints_list, putative_corr_idxs_dict + + def generate_correspondences_inline( + self, + images: List[Image], + visibility_graph: VisibilityGraph, + ) -> Tuple[List[Keypoints], Dict[Tuple[int, int], np.ndarray]]: + """Inline, no-Dask variant of ``generate_correspondences``. + + Detection runs once per image and matching once per pair, in plain Python loops calling the + (cache-backed) detector-descriptor and matcher directly. Computed-and-discarded item by item, so + peak memory is bounded to one cluster's features rather than a worker-resident pile of every + feature/correspondence future. Mirrors the synchronous style of ``ColmapCorrespondenceGenerator``. + + Args: + images: Materialized images indexed by position (``images[i]`` is image ``i``). + visibility_graph: Image pairs ``(i1, i2)`` to match; indices reference ``images``. + + Returns: + keypoints_list: one ``Keypoints`` per image, in index order. + putative_corr_idxs_dict: per-pair putative correspondence indices. + """ + num_images = len(images) + logger.info("🔵 [frontend] Detecting + describing features on %d images...", num_images) + det_start = time.time() + features: List[Tuple[Keypoints, np.ndarray]] = [] + for i, image in enumerate(images): + features.append(self._detector_descriptor.detect_and_describe(image)) + if (i + 1) % 250 == 0 or (i + 1) == num_images: + logger.info("🔵 [frontend] detection %d/%d images (%.0fs)", i + 1, num_images, time.time() - det_start) + keypoints_list = [keypoints for keypoints, _ in features] + + num_pairs = len(visibility_graph) + logger.info("🔵 [frontend] Matching %d pairs...", num_pairs) + match_start = time.time() + putative_corr_idxs_dict: Dict[Tuple[int, int], np.ndarray] = {} + for p, (i1, i2) in enumerate(visibility_graph): + keypoints_i1, descriptors_i1 = features[i1] + keypoints_i2, descriptors_i2 = features[i2] + putative_corr_idxs_dict[(i1, i2)] = self._matcher.match( + keypoints_i1, + keypoints_i2, + descriptors_i1, + descriptors_i2, + im_shape_i1=images[i1].shape, + im_shape_i2=images[i2].shape, + ) + if (p + 1) % 5000 == 0 or (p + 1) == num_pairs: + elapsed = time.time() - match_start + rate = (p + 1) / elapsed if elapsed > 0 else 0.0 + eta = (num_pairs - (p + 1)) / rate if rate > 0 else 0.0 + logger.info( + "🔵 [frontend] matching %d/%d pairs (%.0fs, %.0f pair/s, ETA %.0fs)", + p + 1, num_pairs, elapsed, rate, eta, + ) + + return keypoints_list, putative_corr_idxs_dict diff --git a/gtsfm/frontend/detector_descriptor/colmap_sift.py b/gtsfm/frontend/detector_descriptor/colmap_sift.py index de9de8ee8..6a9781b7f 100644 --- a/gtsfm/frontend/detector_descriptor/colmap_sift.py +++ b/gtsfm/frontend/detector_descriptor/colmap_sift.py @@ -41,7 +41,12 @@ def detect_and_describe(self, image: Image) -> Tuple[Keypoints, np.ndarray]: # Create pycolmap object every time as the object is not pickle-able. # Note: cannot use SiftGPU as wheels are not built with CUDA support. - options = pycolmap.SiftExtractionOptions(max_num_features=self.max_keypoints) + # num_threads=1: this runs once per image under Dask worker-level parallelism, so the parallelism + # comes from running many images concurrently across workers -- NOT from each SIFT grabbing every + # core. The pycolmap default (num_threads=-1) makes every worker's extractor spawn ncores threads, + # so N workers oversubscribe the box by ~N x and thrash to a standstill (looks like a hang). Images + # are downscaled to the loader's max_resolution, so single-threaded extraction is fast regardless. + options = pycolmap.SiftExtractionOptions(max_num_features=self.max_keypoints, num_threads=1) sift_obj = pycolmap.Sift(options) # Extract features. diff --git a/gtsfm/frontend/geometry_transformer.py b/gtsfm/frontend/geometry_transformer.py index 0eb75b920..fbdab47d3 100644 --- a/gtsfm/frontend/geometry_transformer.py +++ b/gtsfm/frontend/geometry_transformer.py @@ -64,3 +64,23 @@ def predict( Returns: A :class:`GeometryTransformerOutput` containing poses, depths, points, etc. """ + + def load_image_batch(self, loader, indices, mode: str = "crop"): + """Load + preprocess a batch of images for this geometry model. + + Image preprocessing (resolution, patch alignment, cropping) is model-specific, and the + per-pixel depth lookup downstream indexes the model's depth map using the ``original_coords`` + returned here — so the loader MUST match the resolution the model outputs. The default is the + VGGT loader; models with different preprocessing (e.g. VGGT-Omega) override this. + + Args: + loader: GTSFM loader providing ``get_image``. + indices: image indices to load. + mode: preprocessing mode (VGGT: ``crop``/``pad``). Implementations may ignore it. + + Returns: + ``(image_batch, original_coords)`` — the padded image tensor and (N, 6) crop/pad metadata. + """ + from gtsfm.frontend.vggt_geometry_transformer import load_image_batch_vggt_loader + + return load_image_batch_vggt_loader(loader, indices, mode=mode) diff --git a/gtsfm/frontend/matcher/torch_twoway_matcher.py b/gtsfm/frontend/matcher/torch_twoway_matcher.py new file mode 100644 index 000000000..21313f3a2 --- /dev/null +++ b/gtsfm/frontend/matcher/torch_twoway_matcher.py @@ -0,0 +1,99 @@ +"""GPU/torch two-way (mutual nearest neighbor) matcher with Lowe ratio test. + +Drop-in alternative to TwoWayMatcher: identical semantics (mutual NN + ratio test on L2 +distances), implemented as one distance GEMM + topk per direction. On an idle datacenter GPU a +full 8192x8192 SIFT pair costs ~1-2 ms vs ~2-4 s for single-threaded OpenCV BFMatcher — the +matching stage drops from hours to minutes. Falls back to CPU torch (threads pinned to 1; dask +workers provide the parallelism) when CUDA is unavailable. +""" + +from typing import Optional, Tuple + +import numpy as np +import torch + +from gtsfm.common.keypoints import Keypoints +from gtsfm.frontend.matcher.matcher_base import MatcherBase + + +class TorchTwoWayMatcher(MatcherBase): + """Mutual-NN + ratio-test matcher on torch (CUDA when available).""" + + def __init__(self, ratio_test_threshold: Optional[float] = 0.8, device: Optional[str] = None) -> None: + super().__init__() + self._ratio_test_threshold = ratio_test_threshold + self._device = device # None => auto (cuda if available else cpu), resolved lazily per process + + def __repr__(self) -> str: + return f"TorchTwoWayMatcher(ratio_test_threshold={self._ratio_test_threshold})" + + def _resolve_device(self) -> torch.device: + if self._device is not None: + return torch.device(self._device) + return torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu") + + def match( + self, + keypoints_i1: Keypoints, + keypoints_i2: Keypoints, + descriptors_i1: np.ndarray, + descriptors_i2: np.ndarray, + im_shape_i1: Tuple[int, int], + im_shape_i2: Tuple[int, int], + ) -> np.ndarray: + if descriptors_i1 is None or descriptors_i2 is None: + return np.array([], dtype=np.uint32) + if descriptors_i1.size == 0 or descriptors_i2.size == 0: + return np.array([], dtype=np.uint32) + + valid_idx_i1 = np.nonzero(~(np.isnan(descriptors_i1).any(axis=1)))[0] + valid_idx_i2 = np.nonzero(~(np.isnan(descriptors_i2).any(axis=1)))[0] + if len(valid_idx_i1) < 2 or len(valid_idx_i2) < 2: + return np.array([], dtype=np.uint32) + + device = self._resolve_device() + if device.type == "cpu": + torch.set_num_threads(1) # dask workers are the parallelism (same rule as cv2/pycolmap) + + with torch.no_grad(): + d1 = torch.from_numpy(np.ascontiguousarray(descriptors_i1[valid_idx_i1], dtype=np.float32)).to(device) + d2 = torch.from_numpy(np.ascontiguousarray(descriptors_i2[valid_idx_i2], dtype=np.float32)).to(device) + # Squared L2 via the GEMM identity; monotone in L2 so NN/topk order is identical, and the + # ratio test is applied on true (sqrt'd, clamped) distances to match OpenCV exactly. + sq1 = (d1 * d1).sum(dim=1, keepdim=True) + sq2 = (d2 * d2).sum(dim=1, keepdim=True) + dist2 = sq1 + sq2.T - 2.0 * (d1 @ d2.T) + dist2.clamp_(min=0.0) + + # direction 1->2 + two_12 = torch.topk(dist2, k=min(2, dist2.shape[1]), dim=1, largest=False) + nn12 = two_12.indices[:, 0] + # direction 2->1 + two_21 = torch.topk(dist2, k=min(2, dist2.shape[0]), dim=0, largest=False) + nn21 = two_21.indices[0, :] + + mutual = nn21[nn12] == torch.arange(dist2.shape[0], device=device) + + if self._ratio_test_threshold is not None and dist2.shape[1] >= 2 and dist2.shape[0] >= 2: + # OpenCV TwoWayMatcher ratio-tests EACH direction before the mutual intersection — + # a match must pass Lowe's test both 1->2 and 2->1. + dists_12 = torch.sqrt(two_12.values) + ratio_ok_12 = dists_12[:, 0] <= self._ratio_test_threshold * dists_12[:, 1] + dists_21 = torch.sqrt(two_21.values) # (2, n2): first/second NN distance per d2 col + ratio_ok_21 = dists_21[0, :] <= self._ratio_test_threshold * dists_21[1, :] + keep = mutual & ratio_ok_12 & ratio_ok_21[nn12] + else: + keep = mutual + dists_12 = torch.sqrt(two_12.values) + + rows = torch.nonzero(keep, as_tuple=False).squeeze(1) + if rows.numel() == 0: + return np.array([], dtype=np.uint32) + # sort by match distance (ascending), mirroring the OpenCV path + order = torch.argsort(dists_12[rows, 0]) + rows = rows[order] + cols = nn12[rows] + idx1 = valid_idx_i1[rows.cpu().numpy()] + idx2 = valid_idx_i2[cols.cpu().numpy()] + + return np.column_stack([idx1, idx2]).astype(np.uint32) diff --git a/gtsfm/frontend/matcher/twoway_matcher.py b/gtsfm/frontend/matcher/twoway_matcher.py index 536307cef..16921bc74 100644 --- a/gtsfm/frontend/matcher/twoway_matcher.py +++ b/gtsfm/frontend/matcher/twoway_matcher.py @@ -104,6 +104,11 @@ def __init_opencv_matcher(self) -> cv.DescriptorMatcher: def __perform_matching(self, descriptors_1: np.ndarray, descriptors_2: np.ndarray) -> np.ndarray: """Run the core logic for matching. + NOTE: cv2's BFMatcher parallelizes internally over the GLOBAL OpenCV thread pool, which + defaults to all cores — with N dask workers that is N full-node thread pools thrashing + each other (same oversubscription bug as pycolmap SIFT num_threads=-1, fixed 818f7b4f). + Parallelism comes from dask workers; each matcher task must stay single-threaded. + Args: descriptors_1: descriptors for the 1st image. descriptors_2: descriptors for the 2nd image. @@ -111,6 +116,8 @@ def __perform_matching(self, descriptors_1: np.ndarray, descriptors_2: np.ndarra Returns: indices of the match between two images. """ + cv.setNumThreads(1) # idempotent; must run in the WORKER process (init doesn't re-run there) + match_indices_1to2: Dict[int, int] = self.__perform_oneway_matching(descriptors_1, descriptors_2) match_indices_2to1: Dict[int, int] = self.__perform_oneway_matching(descriptors_2, descriptors_1) @@ -134,7 +141,12 @@ def __perform_oneway_matching(self, descriptors_1: np.ndarray, descriptors_2: np if self._ratio_test_threshold is not None: all_matches = opencv_matcher.knnMatch(descriptors_1, descriptors_2, k=2) - matches = [m1 for m1, m2 in all_matches if m1.distance <= self._ratio_test_threshold * m2.distance] + # knnMatch(k=2) yields <2 candidates per query when the train side has <2 descriptors + # (near-featureless internet images) — those queries cannot pass a ratio test; skip them. + matches = [ + m[0] for m in all_matches + if len(m) == 2 and m[0].distance <= self._ratio_test_threshold * m[1].distance + ] else: matches = opencv_matcher.match(descriptors_1, descriptors_2) diff --git a/gtsfm/frontend/vggt_omega_geometry_transformer.py b/gtsfm/frontend/vggt_omega_geometry_transformer.py new file mode 100644 index 000000000..c39460185 --- /dev/null +++ b/gtsfm/frontend/vggt_omega_geometry_transformer.py @@ -0,0 +1,347 @@ +"""VGGT Omega implementation of the GeometryTransformer abstraction.""" + +from __future__ import annotations + +import sys +from pathlib import Path +from typing import Any, List, Optional, Union + +import numpy as np +import torch +from PIL import Image as PILImage +from torchvision import transforms as TF + +from gtsfm.frontend.geometry_transformer import GeometryTransformer, GeometryTransformerOutput +from gtsfm.utils import logger as logger_utils +from gtsfm.utils import torch as torch_utils + +PathLike = Union[str, Path] + +logger = logger_utils.get_logger() + +# --------------------------------------------------------------------------- +# Submodule / path management +# --------------------------------------------------------------------------- + +REPO_ROOT = Path(__file__).resolve().parents[2] +THIRDPARTY_ROOT = REPO_ROOT / "thirdparty" +VGGT_OMEGA_SUBMODULE_PATH = THIRDPARTY_ROOT / "vggt-omega" +DEFAULT_WEIGHTS_PATH = VGGT_OMEGA_SUBMODULE_PATH / "weights" / "vggt_omega_1b_512.pt" + + +def _ensure_submodule_on_path(path: Path, name: str) -> None: + """Add a vendored thirdparty module to ``sys.path`` if needed.""" + if not path.exists(): + raise ImportError( + f"Required submodule '{name}' not found at {path}. " + "Run 'git submodule update --init --recursive' to fetch it." + ) + path_str = str(path) + if path_str not in sys.path: + sys.path.insert(0, path_str) + + +_ensure_submodule_on_path(VGGT_OMEGA_SUBMODULE_PATH, "vggt_omega") + + +def unproject_depth_map_to_point_map(depth_map: np.ndarray, extrinsic: np.ndarray, intrinsic: np.ndarray) -> np.ndarray: + depth = depth_map[..., 0] + num_frames, height, width = depth.shape + + y, x = np.meshgrid(np.arange(height), np.arange(width), indexing="ij") + x = np.broadcast_to(x[None], (num_frames, height, width)) + y = np.broadcast_to(y[None], (num_frames, height, width)) + + fx = intrinsic[:, 0, 0][:, None, None] + fy = intrinsic[:, 1, 1][:, None, None] + cx = intrinsic[:, 0, 2][:, None, None] + cy = intrinsic[:, 1, 2][:, None, None] + + camera_points = np.stack( + [ + (x - cx) / fx * depth, + (y - cy) / fy * depth, + depth, + ], + axis=-1, + ) + + rotation = extrinsic[:, :3, :3] + translation = extrinsic[:, :3, 3] + points3d: np.ndarray = np.einsum( + "sij,shwj->shwi", + np.transpose(rotation, (0, 2, 1)), + camera_points - translation[:, None, None, :], + ) + return points3d + + +from vggt_omega.models import VGGTOmega +from vggt_omega.utils.pose_enc import encoding_to_camera + +# --------------------------------------------------------------------------- +# Image loading +# --------------------------------------------------------------------------- + +DEFAULT_FIXED_RESOLUTION_VGGT_OMEGA = 512 + + +def load_image_batch_vggt_omega_loader( + loader, + indices: List[int], + mode: str = "balanced", + image_resolution: int = DEFAULT_FIXED_RESOLUTION_VGGT_OMEGA, + patch_size: int = 16, +) -> tuple[torch.Tensor, torch.Tensor]: + """Load loader-resized images and preprocess them for VGGT-Omega. + + ``original_coords`` maps pixels from ``loader.get_image()`` into the final + padded Omega tensor. Each row is ``[left, top, right, bottom, scaled_w, + scaled_h]`` and can be used as:: + + u_omega = u_loader * scaled_w / loader_w - left + v_omega = v_loader * scaled_h / loader_h - top + + Args: + loader: Loader instance providing ``get_image``. + indices: List of image indices to load. + mode: ``balanced` keeps the total token count close to image_resolution**2. + `max_size` resizes the longest side to image_resolution. + image_resolution: Vggt-Omega preprocessing resolution. + patch_size: Vggt-Omega image patch size. + + Returns: + Batched images shaped ``(N, 3, H, W)`` and coordinates shaped ``(N, 6)``. + """ + # checks from vggt-omega + if mode not in ("balanced", "max_size"): + raise ValueError("Mode must be either 'balanced' or 'max_size'") + if image_resolution <= 0: + raise ValueError("image_resolution must be positive") + if patch_size <= 0: + raise ValueError("patch_size must be positive") + if image_resolution % patch_size != 0: + raise ValueError("image_resolution must be divisible by patch_size") + + images: list[torch.Tensor] = [] + transforms: list[tuple[int, int, int, int, int, int, int, int]] = [] + to_tensor = TF.ToTensor() + + for idx in indices: + image = PILImage.fromarray(loader.get_image(idx).value_array).convert("RGB") + loader_w, loader_h = image.size + + crop_x = 0 + crop_y = 0 + crop_w = loader_w + crop_h = loader_h + aspect_ratio = loader_h / max(loader_w, 1) + if aspect_ratio < 0.5: + crop_w = min(loader_w, max(1, int(round(loader_h / 0.5)))) + crop_x = max((loader_w - crop_w) // 2, 0) + elif aspect_ratio > 2.0: + crop_h = min(loader_h, max(1, int(round(loader_w * 2.0)))) + crop_y = max((loader_h - crop_h) // 2, 0) + + image = image.crop((crop_x, crop_y, crop_x + crop_w, crop_y + crop_h)) + cropped_aspect_ratio = crop_h / max(crop_w, 1) + + if mode == "balanced": + num_patches = (image_resolution // patch_size) ** 2 + width_patches = max(1, int(np.round(np.sqrt(num_patches / cropped_aspect_ratio)))) + height_patches = max(1, int(np.round(num_patches / np.sqrt(num_patches / cropped_aspect_ratio)))) + target_w = width_patches * patch_size + target_h = height_patches * patch_size + elif cropped_aspect_ratio >= 1.0: + target_h = image_resolution + target_w = max( + patch_size, + int(np.round((image_resolution / cropped_aspect_ratio) / patch_size)) * patch_size, + ) + else: + target_w = image_resolution + target_h = max( + patch_size, + int(np.round((image_resolution * cropped_aspect_ratio) / patch_size)) * patch_size, + ) + + image = image.resize((target_w, target_h), PILImage.Resampling.BICUBIC) + images.append(to_tensor(image)) + transforms.append((loader_w, loader_h, crop_x, crop_y, crop_w, crop_h, target_w, target_h)) + + batch_h = max(image.shape[1] for image in images) + batch_w = max(image.shape[2] for image in images) + padded_images: list[torch.Tensor] = [] + original_coords: list[list[float]] = [] + + for image, transform in zip(images, transforms): + loader_w, loader_h, crop_x, crop_y, crop_w, crop_h, target_w, target_h = transform + pad_left = (batch_w - target_w) // 2 + pad_right = batch_w - target_w - pad_left + pad_top = (batch_h - target_h) // 2 + pad_bottom = batch_h - target_h - pad_top + padded_images.append( + torch.nn.functional.pad( + image, + (pad_left, pad_right, pad_top, pad_bottom), + mode="constant", + value=1.0, + ) + ) + + resize_x = target_w / crop_w + resize_y = target_h / crop_h + scaled_w = loader_w * resize_x + scaled_h = loader_h * resize_y + + # this is because the existing frontend expects u_omega = u_loader * resize_x - left + # and natural mapping for this transform is u_omega = (u_loader - crop_x) * resize_x + pad_left + left = crop_x * resize_x - pad_left + top = crop_y * resize_y - pad_top + original_coords.append([left, top, left + batch_w, top + batch_h, scaled_w, scaled_h]) + + return torch.stack(padded_images), torch.tensor(original_coords, dtype=torch.float32) + + +# --------------------------------------------------------------------------- +# Model loading +# --------------------------------------------------------------------------- + + +def resolve_weights_path(weights_path: PathLike | None = None) -> Path: + """Return a concrete path to the VGGT Omega checkpoint, validating that it exists.""" + checkpoint = Path(weights_path) if weights_path is not None else DEFAULT_WEIGHTS_PATH + if not checkpoint.exists(): + raise FileNotFoundError( + f"VGGT Omega weights not found at {checkpoint}. Download weights via `scripts/download_model_weights.sh`." + ) + return checkpoint + + +def load_model( + device: torch.device, +): + """Load the VGGT Omega model weights on the requested device.""" + if device.type != "cuda": + raise RuntimeError("VGGT-Omega requires CUDA.") + checkpoint = resolve_weights_path(DEFAULT_WEIGHTS_PATH) + model = VGGTOmega() + model.load_state_dict(torch.load(checkpoint, map_location="cpu")) + + model.eval() + model.to(device) + return model + + +# Module-level singleton so each Dask worker loads the ~1B VGGT-Omega weights only ONCE. When this +# transformer is driven by ClusterVGGTWithFrontend with model_cache_key=False, the optimizer passes +# model=None for every cluster; without this cache that would reload the 1B model per cluster. +_OMEGA_MODEL_CACHE: dict[str, Any] = {} + + +def resolve_cached_model(device: torch.device): + """Return a per-worker cached VGGT-Omega model, loading the weights once per device.""" + key = str(device) + cached = _OMEGA_MODEL_CACHE.get(key) + if cached is None: + logger.info("⏳ Loading VGGT-Omega model weights (worker cache)...") + cached = load_model(device) + _OMEGA_MODEL_CACHE[key] = cached + logger.info("✅ VGGT-Omega model weights loaded and cached.") + return cached + + +# --------------------------------------------------------------------------- +# VggtOmegaGeometryTransformer +# --------------------------------------------------------------------------- + + +class VggtOmegaGeometryTransformer(GeometryTransformer): + """Runs VGGT Omega model inference to predict poses, depths, and dense points.""" + + def __init__(self): + self.resolved_dtype = torch.float32 + + def load_image_batch(self, loader, indices, mode: str = "balanced"): + """Preprocess images with VGGT-Omega's own loader (512px, 16-patch aligned, aspect-cropped). + + Omega's modes are ``balanced``/``max_size`` (not VGGT's ``crop``/``pad``), so the optimizer's + ``input_mode`` does not apply here — anything other than a valid omega mode falls back to + ``balanced``. The ``original_coords`` returned match the build's keypoint->depth mapping, so the + depth lookup stays consistent with omega's depth-map output resolution. + """ + if mode not in ("balanced", "max_size"): + mode = "balanced" + return load_image_batch_vggt_omega_loader(loader, indices, mode=mode) + + def predict( + self, + images: torch.Tensor, + *, + model: Optional[VGGTOmega] = None, + **kwargs: Any, + ) -> GeometryTransformerOutput: + """Run VGGT Omega forward pass and return unified output. + + Args: + images: Tensor shaped ``(N, 3, H, W)``. + + Returns: + class:`GeometryTransformerOutput` with poses, depths, and dense points. + """ + self.resolved_device = torch_utils.default_device() + images = images.to(self.resolved_device, dtype=self.resolved_dtype) + if model is None: + model = resolve_cached_model(self.resolved_device) + else: + model = model.to(self.resolved_device) + assert model is not None + model.eval() + + with torch.inference_mode(): + predictions = model(images) + + extrinsics, intrinsics = encoding_to_camera( + predictions["pose_enc"], + predictions["images"].shape[-2:], + ) + + depth = predictions["depth"] + depth_conf = predictions["depth_conf"] + + dense_points = unproject_depth_map_to_point_map( + depth.squeeze(0).detach().float().cpu().numpy(), + extrinsics.squeeze(0).detach().float().cpu().numpy(), + intrinsics.squeeze(0).detach().float().cpu().numpy(), + ) + + return GeometryTransformerOutput( + device=self.resolved_device, + dtype=self.resolved_dtype, + images=images, + extrinsic=extrinsics.squeeze(0), + intrinsic=intrinsics.squeeze(0), + depth_map=depth.squeeze(0).squeeze(-1), + depth_confidence=depth_conf.squeeze(0), + dense_points=torch.from_numpy(dense_points).to( + self.resolved_device, + dtype=self.resolved_dtype, + ), + ) + + +# --------------------------------------------------------------------------- +# Public API +# --------------------------------------------------------------------------- + +__all__ = [ + "DEFAULT_WEIGHTS_PATH", + "REPO_ROOT", + "THIRDPARTY_ROOT", + "VGGT_OMEGA_SUBMODULE_PATH", + "VggtOmegaGeometryTransformer", + "_ensure_submodule_on_path", + "load_image_batch_vggt_omega_loader", + "load_model", + "resolve_weights_path", +] diff --git a/gtsfm/loader/loader_base.py b/gtsfm/loader/loader_base.py index cb19b3d56..d1cb6dc48 100644 --- a/gtsfm/loader/loader_base.py +++ b/gtsfm/loader/loader_base.py @@ -370,15 +370,18 @@ def get_image_futures(self, client: Client) -> Dict[int, Future]: Dictionary mapping image index -> Future resolving to `Image`. """ workers = [self._input_worker] if self._input_worker else None - future_map: Dict[int, Future] = {} - for idx in range(len(self)): - future_map[idx] = client.submit( - self.get_image, - idx, - workers=workers, - key=f"loader-get-image-{idx}", - ) - return future_map + indices = list(range(len(self))) + # Bulk-submit in a single update-graph message via client.map. A per-index client.submit loop + # over thousands of images starves the client event loop between submits (it cannot drain the + # scheduler's key-in-memory replies), which closes the client<->scheduler comm mid-submission + # on large scenes (e.g. St Peter's ~2500 imgs). + futures = client.map( + self.get_image, + indices, + key=[f"loader-get-image-{idx}" for idx in indices], + workers=workers, + ) + return {idx: future for idx, future in zip(indices, futures)} def get_key_images_as_delayed_map(self, keys: List[int]) -> Dict[int, Delayed]: """Creates a computation graph to fetch images, using the provided keys as identifiers.""" diff --git a/gtsfm/retriever/__init__.py b/gtsfm/retriever/__init__.py index d86e86fcf..1be89f3a7 100644 --- a/gtsfm/retriever/__init__.py +++ b/gtsfm/retriever/__init__.py @@ -5,18 +5,26 @@ # _target_: gtsfm.retriever.JointSimilaritySequential # _target_: gtsfm.retriever.Sequential # _target_: gtsfm.retriever.Similarity +# _target_: gtsfm.retriever.ColmapDB +# _target_: gtsfm.retriever.ColmapDBMegaLoc +from .colmap_db_megaloc_retriever import ColmapDBMegaLocRetriever +from .colmap_db_retriever import ColmapDBRetriever from .exhaustive_retriever import ExhaustiveRetriever from .joint_similarity_sequential_retriever import JointSimilaritySequentialRetriever from .sequential_retriever import SequentialRetriever from .similarity_retriever import SimilarityRetriever +ColmapDB = ColmapDBRetriever +ColmapDBMegaLoc = ColmapDBMegaLocRetriever Exhaustive = ExhaustiveRetriever JointSimilaritySequential = JointSimilaritySequentialRetriever Sequential = SequentialRetriever Similarity = SimilarityRetriever __all__ = [ + "ColmapDB", + "ColmapDBMegaLoc", "Exhaustive", "JointSimilaritySequential", "Sequential", diff --git a/gtsfm/retriever/colmap_db_megaloc_retriever.py b/gtsfm/retriever/colmap_db_megaloc_retriever.py new file mode 100644 index 000000000..f324078b7 --- /dev/null +++ b/gtsfm/retriever/colmap_db_megaloc_retriever.py @@ -0,0 +1,108 @@ +"""Hybrid retriever: keep each image's top-K COLMAP-verified neighbors, ranked by MegaLoc similarity. + +Sparsifies COLMAP's dense vocab-tree view graph (e.g. ~140k pairs on St Peter's) down to a MegaLoc- +selected graph, WITHOUT fragmenting it: for every image we keep its ``num_matched`` most-MegaLoc-similar +neighbors *among the COLMAP-verified ones*. This is a restricted per-image top-K (not a strict +intersection of two independent top-K sets — that collapses when MegaLoc and COLMAP rank neighbors +differently, as they do on repetitive scenes). Every kept pair is COLMAP-verified, so correspondences +still come from the db (via ``ColmapCorrespondenceGenerator``); the graph stays connected and near-full +coverage while the downstream Metis cluster tree shrinks. + +Authors: Kathirvel Gounder +""" + +from pathlib import Path +from typing import List, Optional + +import numpy as np + +import gtsfm.utils.logger as logger_utils +from gtsfm.products.visibility_graph import VisibilityGraph +from gtsfm.retriever.colmap_db_retriever import ColmapDBRetriever +from gtsfm.retriever.retriever_base import RetrieverBase +from gtsfm.retriever.similarity_retriever import SimilarityRetriever, pairs_from_score_matrix + +logger = logger_utils.get_logger() + + +class ColmapDBMegaLocRetriever(RetrieverBase): + """Per-image top-K over COLMAP-verified neighbors, ranked by MegaLoc similarity.""" + + def __init__(self, database_path: str, num_matched: int, min_score: float = 0.0) -> None: + """ + Args: + database_path: path to the COLMAP database.db (features + verified two-view geometries). + num_matched: number of COLMAP-verified neighbors to keep per image, ranked by MegaLoc sim. + min_score: minimum MegaLoc similarity to keep a (COLMAP-verified) neighbor. Default 0.0 = + rank-only (COLMAP already gates geometric quality; MegaLoc just picks WHICH K to keep). + """ + self._colmap_retriever = ColmapDBRetriever(database_path) + self._similarity_retriever = SimilarityRetriever(num_matched=num_matched, min_score=min_score) + self._num_matched = num_matched + self._min_score = min_score + + def __repr__(self) -> str: + return ( + "ColmapDBMegaLocRetriever:\n" + f" num_matched (COLMAP neighbors/image): {self._num_matched}\n" + f" min_score: {self._min_score}\n" + f" {self._colmap_retriever}" + ) + + def get_image_pairs( + self, + global_descriptors: Optional[List[np.ndarray]], + image_fnames: List[str], + plots_output_dir: Optional[Path] = None, + ) -> VisibilityGraph: + """Return each image's top-K most-MegaLoc-similar COLMAP-verified neighbors. + + Args: + global_descriptors: MegaLoc descriptors, one per image (required for the ranking). + image_fnames: file names of the images. + plots_output_dir: directory to save plots to. If None, plots are not saved. + + Returns: + Sparsified visibility graph: sorted (i, j) pairs with i < j. + """ + colmap_pairs = self._colmap_retriever.get_image_pairs( + global_descriptors=None, image_fnames=image_fnames, plots_output_dir=plots_output_dir + ) + + if global_descriptors is None: + logger.warning( + "🗄️🔎 ColmapDBMegaLocRetriever: no global descriptors provided; returning all %d COLMAP " + "pairs unfiltered (set image_pairs_generator.global_descriptor to enable the MegaLoc filter).", + len(colmap_pairs), + ) + return colmap_pairs + + num_images = len(image_fnames) + sim = self._similarity_retriever.compute_similarity_matrix(global_descriptors) + + # Candidate mask: only COLMAP-verified upper-triangular pairs are allowed. Everything else is + # marked invalid so the per-image top-K picks from an image's COLMAP neighbors, not all images. + is_invalid = ~np.triu(np.ones((num_images, num_images), dtype=bool)) + np.fill_diagonal(is_invalid, True) + if colmap_pairs: + ci = np.fromiter((i for i, _ in colmap_pairs), dtype=int, count=len(colmap_pairs)) + cj = np.fromiter((j for _, j in colmap_pairs), dtype=int, count=len(colmap_pairs)) + colmap_mask = np.zeros((num_images, num_images), dtype=bool) + colmap_mask[ci, cj] = True # COLMAP pairs are already i < j + is_invalid |= ~colmap_mask + + pairs: VisibilityGraph = sorted( + pairs_from_score_matrix( + sim, invalid=is_invalid, num_select=self._num_matched, min_score=self._min_score + ) + ) + logger.info( + "🗄️🔎 ColmapDBMegaLocRetriever: %d COLMAP-verified pairs → top-%d per image by MegaLoc " + "(min_score=%.2f) = %d pairs (from %d images).", + len(colmap_pairs), + self._num_matched, + self._min_score, + len(pairs), + num_images, + ) + return pairs diff --git a/gtsfm/retriever/colmap_db_retriever.py b/gtsfm/retriever/colmap_db_retriever.py new file mode 100644 index 000000000..c76868b33 --- /dev/null +++ b/gtsfm/retriever/colmap_db_retriever.py @@ -0,0 +1,102 @@ +"""Retriever that reads image pairs from a COLMAP database's verified two-view geometries. + +For paper-scale scenes (e.g. the large 1DSfM / Phototourism sets) we build the frontend offline with +COLMAP (GPU SIFT extraction + vocab-tree matching), which is far faster and more memory-efficient than +GTSFM's Dask SIFT frontend. This retriever returns exactly the image pairs COLMAP geometrically verified +(the ``two_view_geometries`` table), so the verified pipeline partitions on the db's verified view graph. +Pair it with ``ColmapCorrespondenceGenerator``, which reads the keypoints + inlier matches for these pairs. + +Authors: GTSFM +""" + +import os +import sqlite3 +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +import numpy as np + +import gtsfm.utils.logger as logger_utils +from gtsfm.products.visibility_graph import VisibilityGraph +from gtsfm.retriever.retriever_base import RetrieverBase + +logger = logger_utils.get_logger() + +# COLMAP encodes a pair as pair_id = image_id1 * MAX + image_id2 (image_ids are 1-indexed). +_COLMAP_MAX_IMAGE_ID = 2147483647 + + +def _pair_id_to_image_ids(pair_id: int) -> Tuple[int, int]: + """Decode a COLMAP pair_id back into its two image ids.""" + image_id2 = pair_id % _COLMAP_MAX_IMAGE_ID + image_id1 = (pair_id - image_id2) // _COLMAP_MAX_IMAGE_ID + return int(image_id1), int(image_id2) + + +class ColmapDBRetriever(RetrieverBase): + """Return the geometrically-verified image pairs from a COLMAP ``database.db``.""" + + def __init__(self, database_path: str) -> None: + """ + Args: + database_path: path to the COLMAP database.db (features + verified two-view geometries). + """ + self._database_path = Path(database_path) + + def __repr__(self) -> str: + return f"ColmapDBRetriever(database_path={self._database_path})" + + def get_image_pairs( + self, + global_descriptors: Optional[List[np.ndarray]], + image_fnames: List[str], + plots_output_dir: Optional[Path] = None, + ) -> VisibilityGraph: + """Read verified pairs from the db and map COLMAP image ids -> GTSFM image indices by filename.""" + del global_descriptors, plots_output_dir # unused: pairs come from the db, not descriptors + if not self._database_path.exists(): + raise FileNotFoundError(f"COLMAP database not found: {self._database_path}") + + # GTSFM index keyed by basename. Loader filenames may be relative paths; the db stores names matching + # the image_path the db was built on — basename is the robust common denominator. + idx_by_basename: Dict[str, int] = {os.path.basename(f): i for i, f in enumerate(image_fnames)} + + conn = sqlite3.connect(str(self._database_path)) + try: + # COLMAP image_id -> GTSFM index (by basename match). + colmap_id_to_gtsfm: Dict[int, int] = {} + for colmap_id, name in conn.execute("SELECT image_id, name FROM images"): + idx = idx_by_basename.get(os.path.basename(str(name))) + if idx is not None: + colmap_id_to_gtsfm[int(colmap_id)] = idx + + pairs = set() + n_verified = 0 + for pair_id, rows, config in conn.execute("SELECT pair_id, rows, config FROM two_view_geometries"): + # config 2 = CALIBRATED (E), 3 = UNCALIBRATED (F); rows>0 means inlier matches exist. + if rows is None or int(rows) == 0 or int(config) not in (2, 3): + continue + n_verified += 1 + cid1, cid2 = _pair_id_to_image_ids(int(pair_id)) + i, j = colmap_id_to_gtsfm.get(cid1), colmap_id_to_gtsfm.get(cid2) + if i is None or j is None or i == j: + continue + pairs.add((i, j) if i < j else (j, i)) + finally: + conn.close() + + visibility_graph: VisibilityGraph = sorted(pairs) + logger.info( + "🗄️ ColmapDBRetriever: %d verified pairs in db; %d/%d loader images matched by name → %d GTSFM pairs.", + n_verified, + len(colmap_id_to_gtsfm), + len(image_fnames), + len(visibility_graph), + ) + if len(colmap_id_to_gtsfm) < len(image_fnames): + logger.warning( + "ColmapDBRetriever: %d loader images had NO match in the db by basename — confirm the db was " + "built on the same images/ directory.", + len(image_fnames) - len(colmap_id_to_gtsfm), + ) + return visibility_graph diff --git a/gtsfm/runner.py b/gtsfm/runner.py index fea7fb9f1..670ec692d 100644 --- a/gtsfm/runner.py +++ b/gtsfm/runner.py @@ -17,7 +17,16 @@ from gtsfm.loader.configuration import add_loader_args, build_loader_overrides from gtsfm.utils.configuration import log_full_configuration -dask_config.set({"distributed.scheduler.worker-ttl": None}) +dask_config.set( + { + "distributed.scheduler.worker-ttl": None, + # Tolerate long event-loop stalls (bulk image loads, the in-process global frontend) without + # tearing down the client<->scheduler/worker comms. With the 30s defaults these comms close + # mid-run on large single-node scenes, surfacing as CommClosedError / "lost dependencies". + "distributed.comm.timeouts.connect": "300s", + "distributed.comm.timeouts.tcp": "300s", + } +) logger = logger_utils.get_logger() diff --git a/gtsfm/scene_optimizer.py b/gtsfm/scene_optimizer.py index 3b0ab6bf3..9c01822a9 100644 --- a/gtsfm/scene_optimizer.py +++ b/gtsfm/scene_optimizer.py @@ -9,6 +9,7 @@ from typing import Optional, TypeVar, cast import matplotlib +import numpy as np from dask.delayed import delayed from dask.distributed import Client, Future, performance_report from omegaconf import OmegaConf @@ -97,6 +98,439 @@ def _finalize_io_tasks(*_args: object) -> None: return None +def _load_precomputed_frontend(path: str, num_images: int, expected_dims=None, max_resolution=None): + """Load a precomputed frontend (scripts/convert_wilsonkl_frontend.py output). + + Returns (padded_keypoints_list, v_corr_idxs_dict) — the exact products of the global + verification stage. The 1DSfM benchmark ships the frontend all published methods consumed + (EGs/coords/tracks); consuming it skips retrieval + SIFT + matching + two-view entirely + and makes comparisons to the published table rows share identical two-view inputs. + """ + from gtsfm.common.keypoints import Keypoints + + z = np.load(path) + n = int(z["n_images"]) + if n > num_images: + raise ValueError(f"precomputed frontend covers {n} images but loader has {num_images}") + if max_resolution is not None and "max_resolution" in z and int(z["max_resolution"]) != int(max_resolution): + raise ValueError( + f"precomputed frontend was converted at max_resolution={int(z['max_resolution'])} " + f"but this run uses {int(max_resolution)} — reconvert." + ) + # EXIF-rotation guard: the benchmark's coords are in Bundler's raw pixel frame; our loader + # applies EXIF transpose. Drop images whose loader-frame dims disagree with the converter's + # expectation (their keypoints would be transposed -> silent geometric poison otherwise). + dropped_imgs = set() + if expected_dims is not None and "img_dims" in z: + img_dims = z["img_dims"] + for i, (w, h) in expected_dims.items(): + if i < n and img_dims[i][0] > 0: + if abs(int(img_dims[i][0]) - int(w)) > 2 or abs(int(img_dims[i][1]) - int(h)) > 2: + dropped_imgs.add(i) + if dropped_imgs: + logger.warning( + "📦 Precomputed frontend: dropping %d image(s) whose loader dims mismatch the " + "converted frame (EXIF-rotation / different source files).", len(dropped_imgs) + ) + kp_offsets, kp_flat = z["kp_offsets"], z["kp_flat"] + keypoints_list = [] + for i in range(num_images): + if i < n and i not in dropped_imgs: + a, b = int(kp_offsets[i]), int(kp_offsets[i + 1]) + keypoints_list.append(Keypoints(coordinates=kp_flat[a:b].astype(np.float64))) + else: + keypoints_list.append(Keypoints(coordinates=np.zeros((0, 2)))) + edges, corr_offsets, corr_flat = z["edges"], z["corr_offsets"], z["corr_flat"] + v_corr_idxs_dict = {} + for k in range(len(edges)): + i1, i2 = int(edges[k][0]), int(edges[k][1]) + if i1 in dropped_imgs or i2 in dropped_imgs: + continue + a, b = int(corr_offsets[k]), int(corr_offsets[k + 1]) + v_corr_idxs_dict[(i1, i2)] = corr_flat[a:b] + return keypoints_list, v_corr_idxs_dict + + +def _run_post_merge_retriangulation( + scene: GtsfmData, + tracks_2d: list, + options: MergingOptions, + trackless_cameras: Optional[dict] = None, +) -> GtsfmData: + """Re-triangulate verified multiview 2D tracks against the merged cameras, then bundle-adjust. + + Mirrors the cluster-level retri stage (`_run_cluster_ba`): builds a fresh sparse structure from + the verified correspondences (discarding the VGGT-depth points), runs BA -- which jointly refines + structure AND the merged (VGGT-predicted -> per-cluster-BA'd -> merged -> merge-BA'd) poses -- then + filters by reprojection error. Designed to run as a single dask task on a worker. + + If `trackless_cameras` is provided (good-pose cameras the merge dropped for lack of a VGGT-depth + track), their merged poses are injected so the geometric retriangulation -- which is depth-confidence + INDEPENDENT -- can re-triangulate their existing global 2D tracks and recover them. Cameras that + still fail to gain a >=3-view track are excluded from BA and dropped by the final filter. + """ + from gtsfm.bundle.bundle_adjustment import multi_view_retriangulate_from_2d_tracks + + if trackless_cameras: + injected = [] + for i, cam in trackless_cameras.items(): + if cam is not None and scene.get_camera(i) is None: + scene.add_camera(i, cam) # local op on this worker's copy of the merged scene + injected.append(i) + logger.info( + "♻️ Post-merge retri: injected %d trackless good-pose camera(s) for geometric recovery: %s", + len(injected), + sorted(injected), + ) + + ba_options = options.ba_options + if options.retri_free_ba: + # Fully-free final solve (GLOMAP-style): the calibration/pose priors that protect the + # incremental merges throttle the full-scene BA — dropped here. Robustness (robust kernel / + # GNC) is inherited from ba_options unchanged. + from dataclasses import replace as _dc_replace + + ba_options = _dc_replace( + ba_options, + use_calibration_prior=False, + use_pose_prior_all_cameras=False, + ) + logger.info("🔓 Post-merge retri BA running FREE (no calib/pose priors).") + + refined = scene + for retri_iter in range(max(1, options.retri_iterations)): + retri = multi_view_retriangulate_from_2d_tracks(refined, tracks_2d) + if retri.number_tracks() == 0: + logger.warning("Post-merge retriangulation produced no tracks; keeping previous scene.") + return refined if retri_iter > 0 else scene + optimizer = ba_options.to_optimizer(min_track_length=options.min_track_length) + refined, _ = optimizer.run_simple_ba(retri) + refined = refined.filter_landmark_measurements( + options.post_ba_max_reproj_error, + options.min_track_length, + # When recovering trackless cameras, force-drop any that still fail to triangulate (decoupled + # from keep_all_cameras) so unrecovered cams don't linger as pose-only and skew the AUC denom. + retain_cameras_without_tracks=(False if trackless_cameras else options.keep_all_cameras), + ) + logger.info( + "🔁 Retri iteration %d/%d: %d tracks after BA + filter.", + retri_iter + 1, + max(1, options.retri_iterations), + refined.number_tracks(), + ) + # Carry the merged scene's full image set (incl. unregistered images) so the retri pose metrics + # use the SAME all-images GT denominator as the merged metrics; otherwise the retri "overall" AUC + # is computed over only its registered cameras and collapses into "constructed-only". + refined._image_info = scene._clone_image_info() + + if trackless_cameras: + recovered_set = set(refined.get_valid_camera_indices()) + for i in sorted(trackless_cameras): + touching = [t for t in tracks_2d if any(m.i == i for m in t.measurements)] + max_views = max((t.number_measurements() for t in touching), default=0) + n_retri = len(refined.get_measurements_for_camera(i)) # 0 if dropped; <15 keeps the merged pose + logger.info( + "♻️ Retri recovery [cam %d]: recovered=%s, retri_tracks=%d (global tracks touching=%d, " + "max track views=%d) [<15 retri_tracks => kept merged pose, excluded from retri BA]", + i, + i in recovered_set, + n_retri, + len(touching), + max_views, + ) + + return refined + + +# --------------------------------------------------------------------------- +# Post-root-merge BOUNDARY RECOVERY (offline-validated exp5 recipe). +# +# The root merge can only keep cameras some cluster reconstructed AND some Sim3 seat carried in; island +# cameras (clusters the MERGE_GUARD dropped, cams the per-cluster BA lost) end up with global-track +# measurements but no pose. The recovery: (a) read the merged scene's own 3D as {gid: xyz} via the +# majority-vote track identity, (b) DLT-triangulate every "boundary" global track that has >=3 posed +# views (<3px mean reproj), (c) RANSAC-DLT resect unposed cameras against the UNION cluster-3D ∪ +# fresh-3D, iterating b<->c so newly posed cams enable new triangulations, (d) one BA over the +# augmented scene. The union in (c) is essential: offline, an island camera saw 508 anchors in +# cluster-3D vs 95 in fresh-only triangulation — the union is what made resection work. +# --------------------------------------------------------------------------- + + +def _dlt_triangulate( + measurements: list, + posed: dict, + intrinsics: dict, +) -> tuple[Optional[np.ndarray], Optional[float]]: + """DLT multiview triangulation from posed cams; returns (X, mean reproj px) or (None, None). + + Faithful port of the offline-validated triangulate(): ``measurements`` is [(cam, u, v), ...], + ``posed[i] = (R_w2c, t_w2c)`` (COLMAP convention), ``intrinsics[i] = (f, k1, k2, px, py)`` with only + f/px/py used (undistorted pinhole model, same as the offline replay). + """ + A = [] + for i, u, v in measurements: + Rm, tr = posed[i] + f, _, _, px, py = intrinsics[i] + xn, yn = (u - px) / f, (v - py) / f + P = np.hstack([Rm, tr[:, None]]) + A.append(xn * P[2] - P[0]) + A.append(yn * P[2] - P[1]) + _, _, Vt = np.linalg.svd(np.array(A)) + Xh = Vt[-1] + if abs(Xh[3]) < 1e-12: + return None, None + X = Xh[:3] / Xh[3] + errs = [] + for i, u, v in measurements: + Rm, tr = posed[i] + f, _, _, px, py = intrinsics[i] + p = Rm @ X + tr + if p[2] <= 1e-9: # behind a camera -> reject + return None, None + errs.append(np.hypot(p[0] / p[2] * f + px - u, p[1] / p[2] * f + py - v)) + return X, float(np.mean(errs)) + + +def _ransac_dlt_resect( + observations: list, + intrinsics_entry: tuple, + struct: dict, + iters: int = 800, + min_inl: int = 10, + px_thresh: float = 4.0, +) -> Optional[tuple[np.ndarray, np.ndarray, np.ndarray, int]]: + """RANSAC-DLT resection of one camera against {gid: xyz}; returns (R_w2c, t_w2c, center, n_inl) or None. + + Faithful port of the offline-validated resect(): ``observations`` is this camera's global-track + measurements [(gid, u, v), ...]; only observations whose gid is in ``struct`` participate. + """ + P3, P2 = [], [] + for t, uu, vv in observations: + X3 = struct.get(int(t)) + if X3 is not None: + P3.append(X3) + P2.append((uu, vv)) + if len(P3) < min_inl: + return None + f, _, _, px0, py0 = intrinsics_entry + X = np.array(P3) + x = (np.array(P2) - [px0, py0]) / f + rng = np.random.default_rng(0) + best, best_inl = -1, None + + def dlt(Xs, xs): + A = [] + for (a, b, c), (xn, yn) in zip(Xs, xs): + A.append([a, b, c, 1, 0, 0, 0, 0, -xn * a, -xn * b, -xn * c, -xn]) + A.append([0, 0, 0, 0, a, b, c, 1, -yn * a, -yn * b, -yn * c, -yn]) + _, _, Vt = np.linalg.svd(np.array(A)) + P = Vt[-1].reshape(3, 4) + U, S, Vt2 = np.linalg.svd(P[:, :3]) + d = np.sign(np.linalg.det(U @ Vt2)) + Rm = U @ np.diag([1, 1, d]) @ Vt2 + sc = (S * [1, 1, d]).mean() + if abs(sc) < 1e-12: + return None + t = P[:, 3] / sc + if np.median((Rm @ X.T).T[:, 2] + t[2]) < 0: # cheirality: majority of points must be in front + Rm, t = -Rm, -t + return Rm, t + + for _ in range(iters): + idx = rng.choice(len(X), 6, replace=False) + fit = dlt(X[idx], x[idx]) + if fit is None: + continue + Rm, t = fit + pr = (Rm @ X.T).T + t + e = np.linalg.norm(pr[:, :2] / np.maximum(pr[:, 2:3], 1e-9) - x, axis=1) + inl = (pr[:, 2] > 1e-9) & (e < px_thresh / f) + if inl.sum() > best: + best, best_inl = int(inl.sum()), inl + if best < min_inl: + return None + fit = dlt(X[best_inl], x[best_inl]) + if fit is None: + return None + Rm, t = fit + return Rm, t, -Rm.T @ t, best + + +def _calibration_to_tuple(cal) -> tuple[float, float, float, float, float]: + """(f, k1, k2, px, py) from a gtsam calibration (k1/k2 default 0 for non-Bundler models).""" + k1 = float(cal.k1()) if hasattr(cal, "k1") else 0.0 + k2 = float(cal.k2()) if hasattr(cal, "k2") else 0.0 + return float(cal.fx()), k1, k2, float(cal.px()), float(cal.py()) + + +def _run_boundary_recovery( + scene: GtsfmData, + tracks_2d: list, + options: MergingOptions, + fallback_intrinsics: Optional[dict] = None, +) -> GtsfmData: + """Boundary triangulation + island-camera resection against the merged root scene, then one BA. + + Runs as a single dask task on a worker (mirrors _run_post_merge_retriangulation). ``scene`` is the + worker-local copy of the root merged scene (mutated in place before BA); ``fallback_intrinsics`` + maps camera index -> (f, k1, k2, px, py) for cameras NOT in the merged scene (global-Fetzer focals + with loader-initial fallback — the same precedence the offline replay used). + """ + from gtsam import Cal3Bundler, PinholeCameraCal3Bundler, Pose3, Rot3, SfmTrack + + from gtsfm import cluster_merging as _cm + + gid_index = _cm._scene_gid_index(scene) + if gid_index is None: + logger.warning( + "🧭 Boundary recovery: merged scene carries no gid-index sidecar " + "(enable_gid_merge_anchoring off?); skipping." + ) + return scene + fallback_intrinsics = fallback_intrinsics or {} + + # Posed cameras of the merged scene, in the COLMAP (R_w2c, t_w2c) convention the DLT helpers use. + posed: dict[int, tuple[np.ndarray, np.ndarray]] = {} + intr: dict[int, tuple[float, float, float, float, float]] = {} + for i in scene.get_valid_camera_indices(): + cam = scene.get_camera(i) + if cam is None: + continue + pose = cam.pose() # gtsam camera pose = cam-to-world + R_w2c = np.asarray(pose.rotation().matrix()).T + posed[i] = (R_w2c, -R_w2c @ np.asarray(pose.translation())) + intr[i] = _calibration_to_tuple(cam.calibration()) + + # (a) gid -> 3D from the merged scene's own tracks (majority-vote identity; first track per gid wins). + struct: dict[int, np.ndarray] = {} + for track in scene.tracks(): + gid = _cm._track_gid(track, gid_index) + if gid >= 0 and gid not in struct: + struct[gid] = np.asarray(track.point3()) + n_cluster3d = len(struct) + + # Per-camera observation index over the global 2D tracks (for resection). + cam_obs: dict[int, list[tuple[int, float, float]]] = {} + for tid, tr in enumerate(tracks_2d): + for m in tr.measurements: + cam_obs.setdefault(int(m.i), []).append((tid, float(m.uv[0]), float(m.uv[1]))) + unposed_candidates = sorted(i for i in cam_obs if i not in posed) + logger.info( + "🧭 Boundary recovery: %d posed cams, %d cluster-3D gids, %d unposed cameras with " + "global-track measurements (%d with intrinsics).", + len(posed), + n_cluster3d, + len(unposed_candidates), + sum(1 for i in unposed_candidates if i in fallback_intrinsics), + ) + + # (b)<->(c): triangulate boundary tracks / resect unposed cameras, iterate (newly posed cams enable + # new triangulations). Cluster 3D wins on gid collision: gids already in `struct` are never + # re-triangulated, so fresh points only AUGMENT the merged structure (the essential union). + boundary_pts: dict[int, np.ndarray] = {} + recovered: dict[int, tuple[np.ndarray, np.ndarray, np.ndarray, int]] = {} + for it in range(3): + new_pts = 0 + for tid, tr in enumerate(tracks_2d): + if tid in struct or tr.number_measurements() < 3: + continue + mp, seen = [], set() + for m in tr.measurements: + i = int(m.i) + if i in posed and i in intr and i not in seen: + mp.append((i, float(m.uv[0]), float(m.uv[1]))) + seen.add(i) + if len(mp) < 3: + continue + X, err = _dlt_triangulate(mp, posed, intr) + if X is not None and err is not None and err < 3.0: + boundary_pts[tid] = X + struct[tid] = X + new_pts += 1 + newly = 0 + for ci in unposed_candidates: + if ci in posed: + continue + k_entry = fallback_intrinsics.get(ci) + if k_entry is None: + continue + r = _ransac_dlt_resect(cam_obs[ci], k_entry, struct) + if r is not None: + Rm, t, C, ninl = r + posed[ci] = (Rm, t) + intr[ci] = k_entry + recovered[ci] = (Rm, t, C, ninl) + newly += 1 + logger.info( + "🧭 Boundary recovery iter %d: +%d triangulated (total boundary %d), " + "+%d cameras resected (total recovered %d, posed %d).", + it, + new_pts, + len(boundary_pts), + newly, + len(recovered), + len(posed), + ) + if newly == 0: + break + + if not recovered and not boundary_pts: + logger.info("🧭 Boundary recovery: nothing to recover (0 boundary tracks, 0 cameras); scene unchanged.") + return scene + + # (d) Add recovered cameras (COLMAP w2c -> gtsam cam-to-world) + boundary tracks restricted to + # posed cameras, then ONE BA over the augmented scene and the standard post-BA reproj filter. + for ci in sorted(recovered): + Rm, t, C, ninl = recovered[ci] + f, k1, k2, px, py = intr[ci] + pose_c2w = Pose3(Rot3(Rm.T), C) # COLMAP w2c -> gtsam camera pose (cam-to-world) + scene.add_camera(ci, PinholeCameraCal3Bundler(pose_c2w, Cal3Bundler(f, k1, k2, px, py))) + logger.info("🧭 Boundary recovery: resected camera %d with %d RANSAC inliers.", ci, ninl) + + final_cams = set(scene.get_valid_camera_indices()) + added_tracks = 0 + for gid in sorted(boundary_pts): + track = SfmTrack(boundary_pts[gid]) + seen = set() + for m in tracks_2d[gid].measurements: + i = int(m.i) + if i in final_cams and i not in seen: + track.addMeasurement(i, np.array([float(m.uv[0]), float(m.uv[1])])) + seen.add(i) + if track.numberMeasurements() >= 3 and scene.add_track(track): + added_tracks += 1 + logger.info( + "🧭 Boundary recovery: triangulated %d boundary tracks (added %d), resected %d cameras; " + "scene now %d cams / %d tracks. Running BA...", + len(boundary_pts), + added_tracks, + len(recovered), + len(final_cams), + scene.number_tracks(), + ) + + try: + optimizer = options.ba_options.to_optimizer(min_track_length=options.min_track_length) + refined, _ = optimizer.run_simple_ba(scene) + refined = refined.filter_landmark_measurements( + options.post_ba_max_reproj_error, + options.min_track_length, + retain_cameras_without_tracks=options.keep_all_cameras, + ) + except Exception as exc: + logger.warning("🧭 Boundary recovery BA failed (%s); returning un-refined augmented scene.", exc) + refined = scene + # Same all-images GT denominator as the merged metrics (see _run_post_merge_retriangulation). + refined._image_info = scene._clone_image_info() + logger.info( + "🧭 Boundary recovery final: %d cameras / %d tracks (was %d cams pre-recovery).", + len(refined.get_valid_camera_indices()), + refined.number_tracks(), + len(final_cams) - len(recovered), + ) + return refined + + class SceneOptimizer: """Wrapper combining different modules to run the whole pipeline on a loader.""" @@ -114,6 +548,35 @@ def __init__( bridge_min_similarity: float = 0.0, bridge_top_k: int = 10, bridge_min_component_size: int = 3, + # --- Verified-viewgraph pipeline: verified-graph partition + post-merge retriangulation --- + use_verified_pipeline: bool = False, + # Attach global-track-id sidecars so merges anchor on identity-matched correspondences (opt-in + # experiment; off = the R3-baseline legacy merge, which produced the most complete ToL structure). + enable_gid_merge_anchoring: bool = False, + # Post-merge retriangulation + BA (structure refinement). Off while re-establishing the + # structure-complete baseline — fewer moving parts; flip back on afterwards. + run_post_merge_retriangulation: bool = True, + # Post-root-merge boundary recovery (offline-validated): triangulate global boundary tracks + # against the merged cameras, RANSAC-DLT resect cameras the merge never placed against the + # cluster-3D ∪ fresh-3D union, iterate, then one BA. Requires the gid sidecar + # (enable_gid_merge_anchoring) to read the merged scene's 3D by global identity. + enable_boundary_recovery: bool = False, + # Export-time low-parallax track filter (0 = off). Tracks whose max pairwise triangulation + # angle is below this are depth-unconstrained: they pass the reprojection filters (reproj + # error is blind to depth error along the ray) yet scatter as fuzz/spray around the + # structure. ToL 120/0.15 audit: 16% of merged tracks sat below 1.5deg (COLMAP's default + # cutoff). Output-side only — merges, poses, and BA never see it. + min_triangulation_angle_deg: float = 0.0, + # Path to a precomputed-frontend npz (scripts/convert_wilsonkl_frontend.py). When set, the + # global frontend (SIFT/matching/two-view) is skipped and these keypoints + verified + # correspondences are used instead — the standard 1DSfM protocol, where all published + # methods consume the benchmark's released view graph and tracks. + precomputed_frontend_path: Optional[str] = None, + # Path to a pair-list file (e.g. a 1DSfM EGs.txt; any lines starting with two integer image + # ids). Skips retrieval and runs OUR full frontend (SIFT/matching/two-view) on exactly these + # pairs — the accuracy of an own-frontend run at a fraction of the matching cost, since + # every pair is known-productive. Ignored when precomputed_frontend_path is set. + precomputed_pairs_path: Optional[str] = None, ) -> None: self.loader = loader self.image_pairs_generator = image_pairs_generator @@ -123,6 +586,13 @@ def __init__( self._bridge_min_similarity = bridge_min_similarity self._bridge_top_k = bridge_top_k self._bridge_min_component_size = bridge_min_component_size + self._use_verified_pipeline = use_verified_pipeline + self._enable_gid_merge_anchoring = enable_gid_merge_anchoring + self._run_post_merge_retriangulation = run_post_merge_retriangulation + self._enable_boundary_recovery = enable_boundary_recovery + self._min_triangulation_angle_deg = min_triangulation_angle_deg + self._precomputed_frontend_path = precomputed_frontend_path + self._precomputed_pairs_path = precomputed_pairs_path # Propagate metric_constructed_only to the cluster optimizer if it supports it. if hasattr(self.cluster_optimizer, "_metric_constructed_only"): setattr(self.cluster_optimizer, "_metric_constructed_only", self._merging_options.metric_constructed_only) @@ -173,6 +643,7 @@ def _schedule_single_cluster(self, context: ClusterContext) -> ClusterExecutionH computation.sfm_result, context.output_paths.plots, context.label, + measurement_gid_index=context.measurement_gid_index, ) io_future: Future = context.client.compute(io_graph) # type: ignore @@ -208,6 +679,208 @@ def run(self, client: Client) -> None: retriever_metrics, visibility_graph, similarity_matrix = self._run_retriever(client, base_output_paths) base_metrics_groups.append(retriever_metrics) image_future_map = self.loader.get_image_futures(client) + one_view_data_dict = self.loader.get_one_view_data_dict() + + # Optional verified-viewgraph pipeline: globally verify the retrieval graph, then + # (a) partition on the verified subgraph and (b) keep global 2D tracks for post-merge retriangulation. + global_tracks_2d: Optional[list] = None + global_refined_intrinsics: Optional[dict] = None + global_v_corr_idxs_dict: Optional[dict] = None + global_keypoints: Optional[list] = None + gid_index_arrays: Optional[tuple] = None # packed (keys, gids, cams) from the global 2D tracks + if self._use_verified_pipeline: + from gtsfm.cluster_optimizer.cluster_mvo import ClusterMVO, _pad_keypoints_list + from gtsfm.two_view_estimator import create_v_corr_idxs_futures + from gtsfm.multi_view_optimizer import get_2d_tracks + from gtsfm.products.visibility_graph import visibility_graph_keys + from gtsfm.utils.graph import get_nodes_in_largest_connected_component + + logger.info("🔎 GTSFM: Global two-view verification over %d retrieval edges...", len(visibility_graph)) + num_images = len(self.loader) + retrieval_edge_count = len(visibility_graph) + # Pass image futures directly; Dask materializes them as a dependency (no nested gather). + image_futures = [image_future_map[idx] for idx in range(num_images)] + + # Reuse the per-cluster frontend chain over the FULL retrieval graph (populates per-pair + # caches, so subsequent per-cluster frontends cache-hit). _run_two_view_estimation already + # filters to result.valid(). + corr_gen = self.cluster_optimizer.correspondence_generator + if self._precomputed_frontend_path: + # Benchmark-released frontend (e.g. 1DSfM EGs/coords/tracks via + # scripts/convert_wilsonkl_frontend.py): the exact two-view inputs the published + # table rows consumed. Skips SIFT + matching + two-view entirely. NOTE: keypoints + # must be in the SAME loader frame (max_resolution) as this run's config. + logger.info( + "📦 Loading precomputed frontend from %s (SIFT/matching/two-view skipped).", + self._precomputed_frontend_path, + ) + _expected_dims = { + int(_i): (2.0 * _ovd.intrinsics.px(), 2.0 * _ovd.intrinsics.py()) + for _i, _ovd in one_view_data_dict.items() + if _ovd.intrinsics is not None + } + padded_keypoints_list, v_corr_idxs_dict = _load_precomputed_frontend( + self._precomputed_frontend_path, + num_images, + expected_dims=_expected_dims, + max_resolution=getattr(self.loader, "_max_resolution", None), + ) + elif getattr(corr_gen, "produces_verified_correspondences", False): + # COLMAP-DB frontend: read the already-verified keypoints + matches straight from the + # database in the main process. No Dask two-view estimation and — crucially — no + # client.gather of all per-edge results, the step that OOM-killed workers at scale. + logger.info("🗄️ Reading verified correspondences from the COLMAP database (Dask frontend skipped).") + keypoints_list, v_corr_idxs_dict = corr_gen.generate_correspondences( + client, image_futures, list(visibility_graph) + ) + padded_keypoints_list = _pad_keypoints_list(keypoints_list, num_images) + else: + # Global two-view over the retrieval graph, returning only v_corr_idxs (each heavy + # per-pair TwoViewResult is dropped the instant it is reduced). run_2view runs + # identically either way (TwoViewEstimatorCacher / DB writes unaffected). Dispatch on + # worker count: + relative_pose_priors = self.loader.get_relative_pose_priors(visibility_graph) or {} + gt_scene_mesh = self.loader.get_gt_scene_trimesh() + try: + num_workers = len(client.scheduler_info()["workers"]) + except Exception: + num_workers = 1 + + if num_workers <= 1: + # Single worker (or unknown): run INLINE in the main process. With no worker to + # lose there is zero scheduler<->worker comm-failure surface — the arrangement that + # otherwise died with "lost dependencies" over multi-hour runs when one monolithic + # task on one worker was dropped. + logger.info("🔵 [frontend] 1 worker → running the global frontend inline (in-process).") + images = client.gather(image_futures) + keypoints_list, putative_corr_idxs_dict, _ = ClusterMVO._run_correspondence_generator( + self.cluster_optimizer.correspondence_generator, list(visibility_graph), images + ) + padded_keypoints_list = _pad_keypoints_list(keypoints_list, num_images) + v_corr_idxs_dict, _ = ClusterMVO._run_two_view_v_corr_idxs( + self.cluster_optimizer.two_view_estimator, + padded_keypoints_list, + putative_corr_idxs_dict, + relative_pose_priors, + gt_scene_mesh, + one_view_data_dict, + ) + else: + # Multiple workers: fan the frontend out across the pool. Correspondence generation + # (per-image detection + per-pair matching) and two-view (chunked) both run in + # parallel; only lean v_corr_idxs is gathered. Chunking (many small tasks, not one + # monolithic task) means a dropped worker recomputes only its chunk, not the whole + # multi-hour frontend. + logger.info("🔵 [frontend] %d workers → running the global frontend in parallel.", num_workers) + keypoints_list, putative_corr_idxs_dict = corr_gen.generate_correspondences( + client, image_futures, list(visibility_graph) + ) + padded_keypoints_list = _pad_keypoints_list(keypoints_list, num_images) + v_corr_idxs_dict = create_v_corr_idxs_futures( + client, + self.cluster_optimizer.two_view_estimator, + padded_keypoints_list, + putative_corr_idxs_dict, + relative_pose_priors, + gt_scene_mesh, + one_view_data_dict, + ) + + verified_graph = sorted(v_corr_idxs_dict.keys()) + all_nodes = visibility_graph_keys(verified_graph) + largest_cc = set(get_nodes_in_largest_connected_component(verified_graph)) if verified_graph else set() + logger.info( + "🔎 Verified graph: %d/%d edges; nodes=%d, largest_cc=%d, dropped_by_partition=%d", + len(verified_graph), + retrieval_edge_count, + len(all_nodes), + len(largest_cc), + len(all_nodes) - len(largest_cc), + ) + visibility_graph = verified_graph + + # Build global 2D tracks (verified correspondences -> union-find tracks) for post-merge retriangulation. + global_tracks_2d = get_2d_tracks(v_corr_idxs_dict, padded_keypoints_list) + logger.info("🔎 Built %d global 2D tracks from verified correspondences.", len(global_tracks_2d)) + + # Persist the global tracks for OFFLINE merge/BA replay (~25MB npz). The COLMAP text exports + # are complete per-node BA problems, but global track IDENTITY (which cluster tracks are the + # same physical point) only exists here — dumping it enables desk-side experiments (merge + # replays, gid-correspondence debugging, cluster-Sim3-averaging prototypes) without PACE runs. + try: + _cams, _us, _vs, _tids = [], [], [], [] + for _tid, _tr in enumerate(global_tracks_2d): + for _m in _tr.measurements: + _cams.append(_m.i) + _us.append(float(_m.uv[0])) + _vs.append(float(_m.uv[1])) + _tids.append(_tid) + np.savez_compressed( + base_output_paths.results / "global_tracks_2d.npz", + cam=np.asarray(_cams, dtype=np.int32), + u=np.asarray(_us, dtype=np.float32), + v=np.asarray(_vs, dtype=np.float32), + track_id=np.asarray(_tids, dtype=np.int32), + ) + logger.info("💾 Saved global tracks for offline replay -> %s", base_output_paths.results / "global_tracks_2d.npz") + del _cams, _us, _vs, _tids + except Exception as _exc: + logger.warning("Failed to save global_tracks_2d.npz (offline replay dump): %s", _exc) + + # Packed (camera, pixel) -> global-track-id arrays. Sliced per cluster below (mirroring the + # per-node context packaging — no global broadcast) so merges can match tracks by GLOBAL + # IDENTITY: same physical point triangulated by two clusters from disjoint cameras. + # Opt-in: without the sidecars the merge runs the legacy (R3-baseline) path. + if self._enable_gid_merge_anchoring: + gid_index_arrays = cluster_merging.build_measurement_gid_arrays(global_tracks_2d) + logger.info( + "🔗 Built global-track-id index: %d measurement keys over %d tracks (%.1f MB packed).", + len(gid_index_arrays[0]), + len(global_tracks_2d), + sum(a.nbytes for a in gid_index_arrays) / 1e6, + ) + + # Reuse the global verified correspondences + keypoints per cluster (skip the per-cluster + # frontend). Plumbed via ClusterContext; clusters subset by their edges and build tracks_2d + # eagerly in the main process (no scatter, no per-cluster two-view re-estimation). + if getattr(self.cluster_optimizer, "reuses_global_correspondences", False): + global_v_corr_idxs_dict = v_corr_idxs_dict + global_keypoints = padded_keypoints_list + + # Global Fetzer calibration: estimate every camera's focal ONCE over the full verified view + # graph (heuristic init, never VGGT focals), supplied to clusters via ClusterContext. Replaces + # the per-cluster calibration, which falls back to VGGT focals for sparsely-connected cameras. + if getattr(self.cluster_optimizer, "uses_global_view_graph_calibration", False): + if getattr(self.cluster_optimizer, "calibration_source", "fetzer") == "exif": + # EXIF passthrough: hand the loader/EXIF intrinsics to every cluster verbatim. + # ToL gold audit: EXIF 1.91% median focal error (unbiased) vs scipy Fetzer 5.26% + # (−4% bias) — and the 1DSfM reference pipeline itself calibrated from EXIF. + # Same frozen-K coherence as Fetzer (bit-identical per camera across clusters). + global_refined_intrinsics = { + idx: ovd.intrinsics + for idx, ovd in one_view_data_dict.items() + if ovd.intrinsics is not None + } + logger.info( + "🔭 Global calibration: EXIF passthrough for %d cameras (Fetzer skipped).", + len(global_refined_intrinsics), + ) + else: + from gtsfm.view_graph_estimator.view_graph_calibration import ( + compute_global_view_graph_intrinsics, + ) + + global_refined_intrinsics = client.gather( + client.compute( + delayed(compute_global_view_graph_intrinsics)( + v_corr_idxs_dict, padded_keypoints_list, one_view_data_dict + ) + ) + ) + logger.info( + "🔭 Global Fetzer calibration: refined %d focals over the full verified view graph.", + len(global_refined_intrinsics), + ) # Bridge reconnection: add cross-component edges to reconnect island components. if similarity_matrix is not None and self._bridge_min_similarity > 0: @@ -237,10 +910,47 @@ def run(self, client: Client) -> None: assert self.graph_partitioner is not None, "Graph partitioner is not set up!" cluster_tree = self.graph_partitioner.run(visibility_graph) self.graph_partitioner.log_partition_details(cluster_tree, base_output_paths) + + # --- Offline-replay telemetry (all small; enables desk-side merge/BA/prior experiments on a + # downloaded results folder without re-running the pipeline) --- + try: + import pickle as _pickle + + with open(base_output_paths.results / "cluster_tree.pkl", "wb") as _f: + _pickle.dump(cluster_tree, _f) # exact tree structure (which node merges into which) + except Exception as _exc: + logger.warning("Failed to save cluster_tree.pkl: %s", _exc) + try: + if self._use_verified_pipeline and v_corr_idxs_dict: + _i = np.array([e[0] for e in v_corr_idxs_dict], dtype=np.int32) + _j = np.array([e[1] for e in v_corr_idxs_dict], dtype=np.int32) + _n = np.array([len(v) for v in v_corr_idxs_dict.values()], dtype=np.int32) + np.savez_compressed( + base_output_paths.results / "verified_graph.npz", i=_i, j=_j, num_inliers=_n + ) # the verified view graph: per-edge inlier counts (edge-addition / connectivity studies) + except Exception as _exc: + logger.warning("Failed to save verified_graph.npz: %s", _exc) + try: + import json as _json + + _cal = {} + for _idx, _ovd in one_view_data_dict.items(): + _entry = {} + if _ovd.intrinsics is not None: + _c = _ovd.intrinsics + _entry["initial"] = [_c.fx(), _c.k1(), _c.k2(), _c.px(), _c.py()] + if global_refined_intrinsics and _idx in global_refined_intrinsics: + _c = global_refined_intrinsics[_idx] + _entry["fetzer"] = [_c.fx(), _c.k1(), _c.k2(), _c.px(), _c.py()] + if _entry: + _cal[int(_idx)] = _entry + with open(base_output_paths.results / "calibrations.json", "w") as _f: + _json.dump(_cal, _f) # loader-initial vs global-Fetzer focals (focal/prior experiments) + except Exception as _exc: + logger.warning("Failed to save calibrations.json: %s", _exc) save_retrieval_two_view_metrics(base_output_paths) logger.info("🔥 GTSFM: Scheduling cluster optimizations...") - one_view_data_dict = self.loader.get_one_view_data_dict() merged_scene: Optional[cluster_merging.MergedNodeSummary] = None with performance_report(filename="dask_reports/scene-optimizer.html"): @@ -261,6 +971,18 @@ def to_context(path: tuple[int, ...], visibility_graph: VisibilityGraph) -> Clus cluster_path=path, label=cluster_label(path), visibility_graph=visibility_graph, + global_refined_intrinsics=global_refined_intrinsics, + global_v_corr_idxs_dict=global_v_corr_idxs_dict, + global_keypoints=global_keypoints, + # Per-cluster slice of the gid index (only this node's cameras) — lean per-node + # packaging, mirrors the context pattern; None disables ID-matching gracefully. + measurement_gid_index=( + cluster_merging.slice_gid_index( + *gid_index_arrays, {k for edge in visibility_graph for k in edge} + ) + if gid_index_arrays is not None + else None + ), ) context_tree = cluster_tree.map_with_path(to_context) @@ -288,10 +1010,12 @@ def merge_fn( export_tree = cluster_merging.schedule_exports(client, handles_tree, merged_future_tree) summary_tree = cluster_merging.schedule_summaries(client, merged_future_tree) root_merge_summary: Optional[cluster_merging.MergedNodeSummary] = None - for handle_node, summary_node, export_node in zip( + root_merged_result: Optional[cluster_merging.MergedNodeResult] = None + for handle_node, summary_node, export_node, merged_node in zip( PreOrderIter(handles_tree), PreOrderIter(summary_tree), PreOrderIter(export_tree), + PreOrderIter(merged_future_tree), ): handle = handle_node.value summary_future = summary_node.value @@ -306,6 +1030,9 @@ def merge_fn( base_metrics_groups.append(merged_summary.metrics) base_metrics_groups.append(merged_summary.pre_ba_metrics) root_merge_summary = merged_summary + if self._use_verified_pipeline: + # Materialize the full root reconstruction (idempotent; already computed for the summary). + root_merged_result = merged_node.value.result() else: merged_summary = summary_future.result() metrics_groups.append(merged_summary.metrics) @@ -315,6 +1042,126 @@ def merge_fn( logger.info("🔥 GTSFM: Running cluster optimization and merging...") merged_scene = root_merge_summary + # Post-merge retriangulation (structure refinement): rebuild sparse structure from the + # globally-verified tracks against the merged poses, then BA. Written as a separate output. + if ( + self._use_verified_pipeline + and self._run_post_merge_retriangulation + and global_tracks_2d + and root_merged_result is not None + and root_merged_result.scene is not None + ): + logger.info( + "🔧 GTSFM: Post-merge retriangulation on %d global 2D tracks vs merged poses...", + len(global_tracks_2d), + ) + refined: GtsfmData = client.submit( + _run_post_merge_retriangulation, + root_merged_result.scene, + global_tracks_2d, + self._merging_options, + root_merged_result.trackless_cameras, + pure=False, + ).result() + retri_dir = base_output_paths.results / "merged_retriangulated" + retri_dir.mkdir(parents=True, exist_ok=True) + refined.export_as_colmap_text(retri_dir) + logger.info( + "🔧 Retriangulated scene: %d images, %d tracks → %s", + refined.number_images(), + refined.number_tracks(), + retri_dir, + ) + base_metrics_groups.append( + cluster_merging.compute_merging_metrics( + refined, + cameras_gt=cameras_gt, + metric_constructed_only=self._merging_options.metric_constructed_only, + suffix="_retriangulated", + ) + ) + + # Post-root-merge boundary recovery: DLT-triangulate boundary global tracks against the + # merged cameras, RANSAC-DLT resect unposed cameras against cluster-3D ∪ fresh-3D, + # iterate, then one BA. Runs as a single dask task on a worker (like the retri stage). + if ( + self._use_verified_pipeline + and self._enable_boundary_recovery + and global_tracks_2d + and root_merged_result is not None + and root_merged_result.scene is not None + ): + # Intrinsics for cameras NOT in the merged scene: global-Fetzer focals, falling + # back to the loader-initial intrinsics (same precedence as the offline replay). + boundary_intrinsics: dict[int, tuple] = {} + for _idx, _ovd in one_view_data_dict.items(): + _cal = None + if global_refined_intrinsics and _idx in global_refined_intrinsics: + _cal = global_refined_intrinsics[_idx] + elif _ovd.intrinsics is not None: + _cal = _ovd.intrinsics + if _cal is not None: + boundary_intrinsics[int(_idx)] = _calibration_to_tuple(_cal) + logger.info( + "🧭 GTSFM: Boundary recovery on %d global 2D tracks vs the merged root scene...", + len(global_tracks_2d), + ) + recovered_scene: GtsfmData = client.submit( + _run_boundary_recovery, + root_merged_result.scene, + global_tracks_2d, + self._merging_options, + boundary_intrinsics, + pure=False, + ).result() + recovery_dir = base_output_paths.results / "merged_boundary_recovered" + recovery_dir.mkdir(parents=True, exist_ok=True) + recovered_scene.export_as_colmap_text(recovery_dir) + logger.info( + "🧭 Boundary-recovered scene: %d images, %d cameras, %d tracks → %s", + recovered_scene.number_images(), + len(recovered_scene.get_valid_camera_indices()), + recovered_scene.number_tracks(), + recovery_dir, + ) + base_metrics_groups.append( + cluster_merging.compute_merging_metrics( + recovered_scene, + cameras_gt=cameras_gt, + metric_constructed_only=self._merging_options.metric_constructed_only, + suffix="_boundary_recovered", + ) + ) + + # Export-time low-parallax cleanup: write an angle-filtered copy of the final merged + # scene alongside merged/ (the unfiltered export stays for A/B). Poses/merges untouched. + if ( + self._use_verified_pipeline + and self._min_triangulation_angle_deg > 0 + and root_merged_result is not None + and root_merged_result.scene is not None + ): + angle_filtered = cluster_merging.filter_tracks_by_triangulation_angle( + root_merged_result.scene, self._min_triangulation_angle_deg + ) + angle_dir = base_output_paths.results / "merged_anglefiltered" + angle_dir.mkdir(parents=True, exist_ok=True) + angle_filtered.export_as_colmap_text(angle_dir) + logger.info( + "🔭 Angle-filtered scene (>=%.1f deg): %d tracks → %s", + self._min_triangulation_angle_deg, + angle_filtered.number_tracks(), + angle_dir, + ) + base_metrics_groups.append( + cluster_merging.compute_merging_metrics( + angle_filtered, + cameras_gt=cameras_gt, + metric_constructed_only=self._merging_options.metric_constructed_only, + suffix="_anglefiltered", + ) + ) + if merged_scene is not None and merged_scene.merge_success: logger.info( "Merged scene contains %d images and %d tracks.", @@ -342,6 +1189,32 @@ def merge_fn( def _run_retriever( self, client: Client, output_paths: OutputPaths ) -> tuple[GtsfmMetricsGroup, VisibilityGraph, Optional[object]]: + # Precomputed pair list (e.g. a 1DSfM EGs.txt): skip retrieval entirely and hand the + # frontend exactly the benchmark's productive pairs — full-quality SIFT/matching/two-view + # runs on only these. Any text file whose lines start with two integer image ids works. + if self._precomputed_pairs_path: + pairs = set() + n_img = len(self.loader) + with open(self._precomputed_pairs_path) as f: + for line in f: + p = line.split() + if len(p) >= 2: + try: + a, b = int(p[0]), int(p[1]) + except ValueError: + continue + if a != b and 0 <= a < n_img and 0 <= b < n_img: + pairs.add((min(a, b), max(a, b))) + visibility_graph = sorted(pairs) + logger.info( + "📦 Precomputed pair list: %d pairs from %s (retrieval skipped).", + len(visibility_graph), self._precomputed_pairs_path, + ) + metrics = GtsfmMetricsGroup( + "retriever_metrics", [GtsfmMetric("num_retrieved_pairs", len(visibility_graph))] + ) + return metrics, visibility_graph, None + # TODO(Frank): refactor to move more of this logic into ImagePairsGenerator retriever_start_time = time.time() batch_size = self.image_pairs_generator._batch_size diff --git a/gtsfm/two_view_estimator.py b/gtsfm/two_view_estimator.py index 1176bf893..13fb4e65f 100644 --- a/gtsfm/two_view_estimator.py +++ b/gtsfm/two_view_estimator.py @@ -13,7 +13,7 @@ from typing import Any, Dict, List, Optional, Tuple import numpy as np -from dask.distributed import Client, Future +from dask.distributed import Client, Future, as_completed from gtsam import Pose3, Rot3, SfmTrack, Unit3 # type: ignore import gtsfm.common.types as gtsfm_types @@ -891,6 +891,194 @@ def apply_two_view_estimator(two_view_estimator: TwoViewEstimator, **kwargs) -> return two_view_result_futures +def create_two_view_results_inline( + two_view_estimator: TwoViewEstimator, + keypoints_list: List[Keypoints], + putative_corr_idxs_dict: AnnotatedGraph[np.ndarray], + relative_pose_priors: Dict[Tuple[int, int], PosePrior], + gt_scene_mesh: Optional[Any], + one_view_data_dict: Dict[int, OneViewData], +) -> AnnotatedGraph[TwoViewResult]: + """Inline (no-Dask) variant of ``create_two_view_estimator_futures``. + + Runs ``run_2view`` for every pair in a plain loop instead of scattering the estimator and submitting a + per-pair task to a nested client. Used by the cluster frontend to avoid the ``worker_client()`` + tasks-launching-tasks pattern. Kwargs mirror ``create_two_view_estimator_futures`` exactly. + """ + two_view_results: AnnotatedGraph[TwoViewResult] = {} + for (i1, i2), putative_corr_idxs in putative_corr_idxs_dict.items(): + view1, view2 = one_view_data_dict[i1], one_view_data_dict[i2] + two_view_results[(i1, i2)] = two_view_estimator.run_2view( + keypoints_i1=keypoints_list[i1], + keypoints_i2=keypoints_list[i2], + putative_corr_idxs=putative_corr_idxs, + camera_intrinsics_i1=view1.intrinsics, + camera_intrinsics_i2=view2.intrinsics, + i2Ti1_prior=relative_pose_priors.get((i1, i2)), + gt_camera_i1=view1.camera_gt, + gt_camera_i2=view2.camera_gt, + gt_scene_mesh=gt_scene_mesh, + i1=i1, + i2=i2, + ) + return two_view_results + + +def create_v_corr_idxs_inline( + two_view_estimator: TwoViewEstimator, + keypoints_list: List[Keypoints], + putative_corr_idxs_dict: AnnotatedGraph[np.ndarray], + relative_pose_priors: Dict[Tuple[int, int], PosePrior], + gt_scene_mesh: Optional[Any], + one_view_data_dict: Dict[int, OneViewData], +) -> AnnotatedGraph[np.ndarray]: + """Memory-bounded variant of ``create_two_view_results_inline`` that retains only ``v_corr_idxs``. + + Runs ``run_2view`` for every pair exactly as the full-result variant, but keeps ONLY the + verified-correspondence index array of each VALID result and drops the full ``TwoViewResult`` + (three ``TwoViewEstimationReport``s + ``putative_corr_idxs``) as soon as its ``v_corr_idxs`` is + extracted. The worker therefore never accumulates all N heavy results at once — the payload that + OOM-killed the global verified-pipeline worker on large scenes. Side effects are identical to the + full variant: ``run_2view`` (and thus the ``TwoViewEstimatorCacher`` / DB writes) still executes + per pair; the ``valid()`` filter mirrors ``ClusterMVO._run_two_view_estimation``. + """ + v_corr_idxs_dict: AnnotatedGraph[np.ndarray] = {} + num_pairs = len(putative_corr_idxs_dict) + start_time = time.time() + for p, ((i1, i2), putative_corr_idxs) in enumerate(putative_corr_idxs_dict.items()): + view1, view2 = one_view_data_dict[i1], one_view_data_dict[i2] + result = two_view_estimator.run_2view( + keypoints_i1=keypoints_list[i1], + keypoints_i2=keypoints_list[i2], + putative_corr_idxs=putative_corr_idxs, + camera_intrinsics_i1=view1.intrinsics, + camera_intrinsics_i2=view2.intrinsics, + i2Ti1_prior=relative_pose_priors.get((i1, i2)), + gt_camera_i1=view1.camera_gt, + gt_camera_i2=view2.camera_gt, + gt_scene_mesh=gt_scene_mesh, + i1=i1, + i2=i2, + ) + if result.valid(): + v_corr_idxs_dict[(i1, i2)] = result.v_corr_idxs + # `result` (with its reports + putative idxs) goes out of scope each iteration -> not retained. + if (p + 1) % 2000 == 0 or (p + 1) == num_pairs: + elapsed = time.time() - start_time + rate = (p + 1) / elapsed if elapsed > 0 else 0.0 + eta = (num_pairs - (p + 1)) / rate if rate > 0 else 0.0 + logger.info( + "🔵 [two-view] %d/%d pairs (%d valid, %.0fs, %.1f pair/s, ETA %.0fs)", + p + 1, num_pairs, len(v_corr_idxs_dict), elapsed, rate, eta, + ) + return v_corr_idxs_dict + + +def create_v_corr_idxs_futures( + client: Client, + two_view_estimator: TwoViewEstimator, + keypoints_list: List[Keypoints], + putative_corr_idxs_dict: AnnotatedGraph[np.ndarray], + relative_pose_priors: Dict[Tuple[int, int], PosePrior], + gt_scene_mesh: Optional[Any], + one_view_data_dict: Dict[int, OneViewData], + chunk_size: Optional[int] = None, + broadcast: bool = True, +) -> AnnotatedGraph[np.ndarray]: + """Parallel, memory-bounded two-view over the Dask worker pool, returning only ``v_corr_idxs``. + + Same computation and result as ``create_v_corr_idxs_inline`` (which it reuses as the per-chunk body), + but the pairs are split into chunks that run concurrently across workers. The shared read-only inputs + (estimator, keypoints, priors, mesh, per-view data) are scattered ONCE — each as a single blob (a bare + ``client.scatter`` of a list/dict would explode it into per-element futures) — so they are not + re-embedded in every task's serialized graph. Each chunk returns only its ``{(i1, i2): v_corr_idxs}`` + sub-dict; the driver gathers the small sub-dicts and merges them. + + Unlike routing the whole frontend through ONE monolithic delayed task on ONE worker (the arrangement + that died with "lost dependencies" when the scheduler<->worker comm hiccupped over a multi-hour run), + a dropped worker here only forces its own chunks to recompute — the rest of the stage is unaffected. + + Args: + client: Dask client whose workers run the chunks. + two_view_estimator: estimator applied per pair (``run_2view``); side effects (cacher/DB) unchanged. + keypoints_list: per-image keypoints (indexed by absolute image index). + putative_corr_idxs_dict: putative correspondences per pair (the work to distribute). + relative_pose_priors: optional per-pair relative pose priors. + gt_scene_mesh: optional GT scene mesh (or ``None``). + one_view_data_dict: per-image intrinsics / GT camera data. + chunk_size: pairs per chunk; ``None`` targets ~4 chunks per worker (keeps all workers busy while + keeping per-task scheduling + transfer overhead negligible). + broadcast: replicate the scattered blobs to every worker up front (default). This is both faster + (no per-task fetch) and MORE robust: scattered data is raw (non-recomputable), so with a single + holder (``broadcast=False``) losing that worker fails the whole stage — the exact failure mode + this parallel frontend exists to avoid. Replicas survive a worker loss and lost chunks recompute. + Set ``broadcast=False`` only when memory-bound on a stable cluster (×N_workers replication saved). + + Returns: + ``{(i1, i2): v_corr_idxs}`` for every VALID pair — identical to ``create_v_corr_idxs_inline``. + """ + pairs = list(putative_corr_idxs_dict.keys()) + num_pairs = len(pairs) + if num_pairs == 0: + return {} + + try: + n_workers = max(1, len(client.scheduler_info()["workers"])) + except Exception: + n_workers = 1 + if chunk_size is None: + target_chunks = max(n_workers * 4, 1) + chunk_size = max(1, (num_pairs + target_chunks - 1) // target_chunks) + + def _scatter_blob(obj: Any) -> Future: + # Wrap in a 1-list so scatter treats the list/dict as ONE object (returns one future) instead of + # scattering each element/value separately. + return client.scatter([obj], broadcast=broadcast)[0] + + estimator_future = client.scatter(two_view_estimator, broadcast=True) + keypoints_future = _scatter_blob(keypoints_list) + priors_future = _scatter_blob(relative_pose_priors) + one_view_data_future = _scatter_blob(one_view_data_dict) + # A None mesh is passed straight through (nothing to scatter); otherwise scatter it as a blob too. + mesh_arg: Any = _scatter_blob(gt_scene_mesh) if gt_scene_mesh is not None else None + + chunks = [ + {p: putative_corr_idxs_dict[p] for p in pairs[start : start + chunk_size]} + for start in range(0, num_pairs, chunk_size) + ] + logger.info( + "🔵 [two-view|parallel] %d pairs over %d worker(s) in %d chunks (~%d pairs/chunk).", + num_pairs, n_workers, len(chunks), chunk_size, + ) + + # pure=False: run_2view is side-effecting (cacher/DB writes) and each chunk is distinct — never + # memoize/dedupe. Positional args follow create_v_corr_idxs_inline's signature exactly. + chunk_futures = [ + client.submit( + create_v_corr_idxs_inline, + estimator_future, + keypoints_future, + chunk, + priors_future, + mesh_arg, + one_view_data_future, + pure=False, + ) + for chunk in chunks + ] + + v_corr_idxs_dict: AnnotatedGraph[np.ndarray] = {} + start_time = time.time() + num_chunks = len(chunk_futures) + for done_count, future in enumerate(as_completed(chunk_futures), start=1): + v_corr_idxs_dict.update(future.result()) + logger.info( + "🔵 [two-view|parallel] %d/%d chunks done (%d valid pairs, %.0fs).", + done_count, num_chunks, len(v_corr_idxs_dict), time.time() - start_time, + ) + return v_corr_idxs_dict + + def get_two_view_reports_summary( two_view_report_dict: AnnotatedGraph[TwoViewEstimationReport], one_view_data_dict: Dict[int, OneViewData], diff --git a/gtsfm/two_view_estimator_cacher.py b/gtsfm/two_view_estimator_cacher.py index df4583317..9bb7ef557 100644 --- a/gtsfm/two_view_estimator_cacher.py +++ b/gtsfm/two_view_estimator_cacher.py @@ -23,7 +23,7 @@ # Number of first K correspondence indices from image pair to use to create cache key. NUM_CORRESPONDENCES_TO_SAMPLE_FOR_HASH = 10 -CACHE_ROOT_PATH = Path(__file__).resolve().parent.parent / "cache" +CACHE_ROOT_PATH = cache_utils.get_cache_root() # honors GTSFM_CACHE_ROOT env var logger = logger_utils.get_logger() diff --git a/gtsfm/utils/cache.py b/gtsfm/utils/cache.py index 1dd05246f..ade34111f 100644 --- a/gtsfm/utils/cache.py +++ b/gtsfm/utils/cache.py @@ -4,6 +4,8 @@ """ import hashlib +import os +from pathlib import Path import numpy as np import torch @@ -11,6 +13,19 @@ from gtsfm.common.image import Image +def get_cache_root() -> Path: + """Root directory for all GTSFM cachers. + + Defaults to /cache; override with the GTSFM_CACHE_ROOT env var — e.g. to point parallel + experiment nodes at one shared scratch cache, or to isolate an experiment's cache entirely. + Read at import time in each process (dask workers inherit the launcher's environment). + """ + override = os.environ.get("GTSFM_CACHE_ROOT") + if override: + return Path(override).expanduser().resolve() + return Path(__file__).resolve().parent.parent.parent / "cache" + + def generate_hash_for_image(image: Image) -> str: """Hash the image using image name, content, and image shape.""" return hashlib.sha1( diff --git a/gtsfm/utils/io.py b/gtsfm/utils/io.py index 9958ab85b..ad2590240 100644 --- a/gtsfm/utils/io.py +++ b/gtsfm/utils/io.py @@ -450,11 +450,25 @@ def read_from_bz2_file(file_path: Path) -> Optional[Any]: def write_to_bz2_file(data: Any, file_path: Path) -> None: - """Writes data using pickle to a compressed file.""" + """Writes data using pickle to a compressed file, ATOMICALLY (tmp + rename). + + Cache entries are written by many workers — and with shared caches, by many nodes — so a + direct write leaves torn/truncated files when a writer dies mid-write or two writers race + (observed in production caches: truncated .pbz2 entries). Writing to a unique tmp file and + os.replace-ing guarantees readers only ever see complete entries; same-key racers harmlessly + overwrite each other with identical content (keys are content hashes). + """ file_path.parent.mkdir(exist_ok=True, parents=True) - pickle.dump(data, BZ2File(file_path, "wb")) - if not file_path.exists(): - logger.debug("Cache file could not be written!") + tmp_path = file_path.with_name(file_path.name + f".tmp.{os.getpid()}") + try: + pickle.dump(data, BZ2File(tmp_path, "wb")) + os.replace(tmp_path, file_path) + except Exception: + logger.exception("Cache file could not be written!") + try: + tmp_path.unlink(missing_ok=True) + except OSError: + pass def save_point_cloud_as_ply(save_fpath: str, points: np.ndarray, rgb: Optional[np.ndarray] = None) -> None: diff --git a/gtsfm/utils/torch.py b/gtsfm/utils/torch.py index 748526189..8cb5d5242 100644 --- a/gtsfm/utils/torch.py +++ b/gtsfm/utils/torch.py @@ -62,7 +62,7 @@ def cal3_bundler_from_intrinsic(matrix: np.ndarray, crop_coords) -> gtsam.Cal3Bu def cal3ds2_from_intrinsic(matrix: np.ndarray, crop_coords) -> gtsam.Cal3_S2: - """Map a 3x3 intrinsic matrix to a Cal3_S2.""" + """Map a 3x3 intrinsic matrix to a Cal3DS2.""" fx = float(matrix[0, 0]) fy = float(matrix[1, 1]) cx = float(matrix[0, 2]) diff --git a/gtsfm/view_graph_estimator/view_graph_calibration.py b/gtsfm/view_graph_estimator/view_graph_calibration.py index 50f33eece..363fb6940 100644 --- a/gtsfm/view_graph_estimator/view_graph_calibration.py +++ b/gtsfm/view_graph_estimator/view_graph_calibration.py @@ -8,18 +8,40 @@ of Devices with Uncorrelated Camera Parameters", WACV 2020. """ -import logging from typing import Dict, List, Optional, Tuple import cv2 +import gtsam import numpy as np from gtsam import Cal3Bundler, Cal3_S2, Cal3DS2 from scipy.optimize import least_squares +from scipy.sparse import coo_matrix + +try: + import poselib # robust F + H estimation for the Fetzer gates + + _HAS_POSELIB = True +except ImportError: # pragma: no cover + _HAS_POSELIB = False + +# Custom gtsam builds (borglab/gtsfm#1115) provide a closed-form SelfCalibrationFactor; standard +# wheels do not. The solver uses it when present and falls back to scipy otherwise — the ToL gold +# A/B showed the GATES carry the improvement (13.25% -> 5.32% median focal error on identical +# inputs), with the solver choice secondary. +# The closed-form focal factor is named SelfCalibrationFactor in current gtsam-develop and +# FetzerFactor in pre-rename builds; older builds also store the focal variables as Vector1 +# instead of double. _solve_focals handles both names and both storage conventions. +_SELFCAL_FACTOR_CLS = getattr(gtsam, "SelfCalibrationFactor", None) or getattr(gtsam, "FetzerFactor", None) +_HAS_SELFCAL_FACTOR = _SELFCAL_FACTOR_CLS is not None +# COLMAP EstimateTwoViewGeometry model-selection threshold: PLANAR if H explains almost as many +# correspondences as F. Planar/panoramic pairs drive the closed-form Fetzer residual to its poles. +_MAX_H_INLIER_RATIO = 0.8 import gtsfm.common.types as gtsfm_types from gtsfm.common.keypoints import Keypoints +from gtsfm.utils.logger import get_logger -logger = logging.getLogger(__name__) +logger = get_logger() def estimate_fundamental_from_correspondences( @@ -43,6 +65,119 @@ def estimate_fundamental_from_correspondences( return F +def f_passes_focal_sanity( + F: np.ndarray, + pp1: np.ndarray, + pp2: np.ndarray, + f_init: float, + lo: float = 0.4, + hi: float = 2.5, + max_resid: float = 0.01, +) -> bool: + """True iff this F implies a well-defined focal: sweeping a shared focal over [lo,hi]*f_init, + the essential-ness residual ((s2/s1-1)^2 + (s3/s1)^2 of K2^T F K1) must reach an INTERIOR + minimum below max_resid. Near-planar/degenerate F's rail to the band edge or never converge — + they poison the joint Fetzer solve (ToL: ungated -12.35% focal bias) while remaining fine for + pose estimation, so this gate only guards calibration.""" + focals = np.linspace(lo * f_init, hi * f_init, 60) + residuals = np.empty(len(focals)) + for i, f in enumerate(focals): + K1 = np.array([[f, 0.0, pp1[0]], [0.0, f, pp1[1]], [0.0, 0.0, 1.0]]) + K2 = np.array([[f, 0.0, pp2[0]], [0.0, f, pp2[1]], [0.0, 0.0, 1.0]]) + s = np.linalg.svd(K2.T @ F @ K1, compute_uv=False) + residuals[i] = (s[1] / s[0] - 1.0) ** 2 + (s[2] / s[0]) ** 2 + best = int(np.argmin(residuals)) + at_band_edge = best <= 1 or best >= len(focals) - 2 + return (not at_band_edge) and bool(residuals[best] <= max_resid) + + +def _estimate_fundamental_robust( + coords_i1: np.ndarray, coords_i2: np.ndarray, threshold_px: float = 2.0 +) -> Tuple[Optional[np.ndarray], Optional[str]]: + """Robust (LO-RANSAC) F via PoseLib + planar-config gate; falls back to the 8-point estimate + when poselib is unavailable. Returns (F, drop_reason) with drop_reason in + {None, 'planar', 'f_fail'}; F is None whenever drop_reason is not None.""" + if not _HAS_POSELIB: + F = estimate_fundamental_from_correspondences(coords_i1, coords_i2) + return (F, None) if F is not None else (None, "f_fail") + try: + F, f_info = poselib.estimate_fundamental( + coords_i1, coords_i2, {"max_epipolar_error": threshold_px}, {} + ) + except Exception: + return None, "f_fail" + F = np.asarray(F, dtype=np.float64) + if F.shape != (3, 3) or not np.all(np.isfinite(F)) or np.allclose(F, 0.0): + return None, "f_fail" + n_F = int(f_info.get("num_inliers", 0)) + try: + _, h_info = poselib.estimate_homography( + coords_i1, coords_i2, {"max_reproj_error": threshold_px}, {} + ) + n_H = int(h_info.get("num_inliers", 0)) + except Exception: + n_H = 0 + if n_H > _MAX_H_INLIER_RATIO * max(n_F, 1): + return None, "planar" + return F, None + + +def _solve_focals(edges, cam_idx_to_var_idx, precomputed, initial_focals, jac_sparsity): + """Joint focal solve: gtsam SelfCalibrationFactor graph (Cauchy 0.01, GLOMAP-matched) when the + custom build provides it, else scipy least_squares with Cauchy loss. Returns (focals, info).""" + if _HAS_SELFCAL_FACTOR: + sorted_cams = sorted(cam_idx_to_var_idx, key=cam_idx_to_var_idx.get) + + def _gtsam_solve(as_vector: bool): + graph = gtsam.NonlinearFactorGraph() + edge_noise = gtsam.noiseModel.Robust.Create( + gtsam.noiseModel.mEstimator.Cauchy.Create(0.01), + gtsam.noiseModel.Isotropic.Sigma(2, 1.0), + ) + for cam1, cam2, F, pp1, pp2 in edges: + graph.add( + _SELFCAL_FACTOR_CLS( + gtsam.symbol("f", cam1), gtsam.symbol("f", cam2), F, pp1, pp2, edge_noise + ) + ) + values = gtsam.Values() + for var_idx, cam in enumerate(sorted_cams): + f0 = float(initial_focals[var_idx]) + values.insert(gtsam.symbol("f", cam), np.array([f0]) if as_vector else f0) + params = gtsam.LevenbergMarquardtParams() + params.setMaxIterations(200) + result = gtsam.LevenbergMarquardtOptimizer(graph, values, params).optimize() + if as_vector: + focals = np.array([float(result.atVector(gtsam.symbol("f", c))[0]) for c in sorted_cams]) + else: + focals = np.array([result.atDouble(gtsam.symbol("f", c)) for c in sorted_cams]) + return focals, graph.error(values), graph.error(result) + + # Current builds store the focal variable as double; pre-rename FetzerFactor builds expect + # Vector1 (optimize() throws a GenericValue-vs-Matrix RuntimeError). Try both. + for as_vector in (False, True): + try: + focals, e0, e1 = _gtsam_solve(as_vector) + return focals, ( + f"gtsam {_SELFCAL_FACTOR_CLS.__name__} LM" + f"{' (Vector1 storage)' if as_vector else ''}, error {e0:.4f} -> {e1:.4f}" + ) + except Exception as exc: + last_exc = exc + logger.warning("%s solve failed (%s); falling back to scipy.", _SELFCAL_FACTOR_CLS.__name__, last_exc) + result = least_squares( + _fetzer_residuals, + x0=initial_focals, + args=(edges, cam_idx_to_var_idx, precomputed), + jac_sparsity=jac_sparsity, + loss="cauchy", + f_scale=0.1, + bounds=(100.0, np.inf), + max_nfev=200, + ) + return result.x, f"scipy {result.message} in {result.nfev} evaluations, cost {result.cost:.4f}" + + def _fetzer_residuals( focal_lengths: np.ndarray, edges: List[Tuple[int, int, np.ndarray, np.ndarray, np.ndarray]], @@ -127,6 +262,7 @@ def calibrate_view_graph( min_focal_ratio: float = 0.5, max_focal_ratio: float = 2.0, max_edge_error: float = 0.5, + use_robust_gates: bool = True, ) -> Tuple[Dict[int, gtsfm_types.CALIBRATION_TYPE], set]: """Refine camera focal lengths via joint Fetzer optimization over all F-matrix edges. @@ -145,9 +281,14 @@ def calibrate_view_graph( Refined intrinsics dict (same keys as initial_intrinsics). Set of edge keys (i1, i2) to remove from the view graph. """ - # Step 1: Estimate F-matrices and collect optimization edges. + # Step 1: Estimate F-matrices and collect optimization edges. With use_robust_gates (default), + # each edge gets a robust PoseLib F plus two admission gates — planar-config (H/F inliers) and + # focal-sanity (interior-minimum sweep). ToL gold A/B: ungated 8-point feeding of planar / + # focal-degenerate edges (56% of that graph!) produced a -12.35% systematic focal bias; the + # gated recipe cut median error 2.5x and removed the bias on identical inputs. edges = [] # (cam_idx1, cam_idx2, F, pp1, pp2) cameras_in_edges = set() + num_planar = num_sanity = num_ffail = 0 for (i1, i2), v_corr_idxs in v_corr_idxs_dict.items(): if v_corr_idxs.shape[0] < min_correspondences: @@ -156,22 +297,46 @@ def calibrate_view_graph( coords_i1 = keypoints[i1].coordinates[v_corr_idxs[:, 0]] coords_i2 = keypoints[i2].coordinates[v_corr_idxs[:, 1]] - F = estimate_fundamental_from_correspondences(coords_i1, coords_i2) - if F is None: - continue - K1 = initial_intrinsics[i1].K() K2 = initial_intrinsics[i2].K() pp1 = np.array([K1[0, 2], K1[1, 2]]) pp2 = np.array([K2[0, 2], K2[1, 2]]) + if use_robust_gates: + F, drop_reason = _estimate_fundamental_robust(coords_i1, coords_i2) + if drop_reason == "planar": + num_planar += 1 + continue + if F is None: + num_ffail += 1 + continue + f_init = 0.5 * (K1[0, 0] + K2[0, 0]) + if not f_passes_focal_sanity(F, pp1, pp2, f_init): + num_sanity += 1 + continue + else: + F = estimate_fundamental_from_correspondences(coords_i1, coords_i2) + if F is None: + num_ffail += 1 + continue + edges.append((i1, i2, F, pp1, pp2)) cameras_in_edges.add(i1) cameras_in_edges.add(i2) + if use_robust_gates: + logger.info( + "View graph calibration gates: %d edges kept | dropped %d planar, %d focal-sanity, " + "%d F-fail (poselib=%s).", + len(edges), num_planar, num_sanity, num_ffail, _HAS_POSELIB, + ) + if not edges: logger.info("View graph calibration: no valid edges, skipping.") - return list(initial_intrinsics), set() + # Return the intrinsics UNCHANGED (as a dict). `list(initial_intrinsics)` would return the + # camera-index KEYS, which downstream code then indexes as if it were the intrinsics dict + # -> Cal3Bundler(pose, ) crash for any cluster with no F-edges. + return dict(initial_intrinsics), set() # Step 2: Set up optimization variables. sorted_cameras = sorted(cameras_in_edges) @@ -196,21 +361,35 @@ def calibrate_view_graph( "pp2_stack": np.array([e[4] for e in edges]), # (n, 2) } - # Step 3: Joint optimization with Cauchy robust loss. - result = least_squares( - _fetzer_residuals, - x0=initial_focals, - args=(edges, cam_idx_to_var_idx, precomputed), - loss="cauchy", - f_scale=0.1, - bounds=(100.0, np.inf), - max_nfev=200, + # Jacobian sparsity: each edge's two residual rows depend ONLY on that edge's two camera focals + # (idx1, idx2) out of all N cameras — a 99.9%-sparse Jacobian. Without this, least_squares builds a + # DENSE numerical Jacobian, perturbing every one of the N focals per iteration (N x batched-SVD), + # which hangs at scale (e.g. 2504 cameras x 117k edges). Passing jac_sparsity lets scipy estimate the + # numerical Jacobian with graph-colored group perturbations + solve it sparsely -> seconds, not hours. + n_edges = len(edges) + n_cams = len(initial_focals) + sparsity_rows = np.repeat(np.arange(2 * n_edges), 2) # rows: [2k, 2k, 2k+1, 2k+1] per edge k + sparsity_cols = np.empty(4 * n_edges, dtype=int) + sparsity_cols[0::4] = precomputed["idx1"] # residual 2k depends on focal idx1 + sparsity_cols[1::4] = precomputed["idx2"] # residual 2k depends on focal idx2 + sparsity_cols[2::4] = precomputed["idx1"] # residual 2k+1 depends on focal idx1 + sparsity_cols[3::4] = precomputed["idx2"] # residual 2k+1 depends on focal idx2 + jac_sparsity = coo_matrix( + (np.ones(4 * n_edges), (sparsity_rows, sparsity_cols)), + shape=(2 * n_edges, n_cams), ) - optimized_focals = result.x + # Step 3: Joint optimization (gtsam SelfCalibrationFactor when the build provides it, else + # scipy Cauchy least squares — see _solve_focals). + optimized_focals, solver_info = _solve_focals( + edges, cam_idx_to_var_idx, precomputed, initial_focals, jac_sparsity + ) - # Step 4: Validate and build refined intrinsics. + # Step 4: Validate and build refined intrinsics. applied_focals tracks the focal actually used + # downstream per camera (optimized when accepted, initial otherwise) so Step 5 scores edges with + # what the pipeline ships, not with discarded optimizer values. refined = dict(initial_intrinsics) + applied_focals = initial_focals.copy() num_refined = 0 num_rejected = 0 @@ -220,6 +399,7 @@ def calibrate_view_graph( ratio = new_focal / old_focal if old_focal > 0 else 0 if min_focal_ratio <= ratio <= max_focal_ratio: + applied_focals[var_idx] = new_focal old_K = initial_intrinsics[cam_idx] # We assume fx == fy here. if isinstance(old_K, Cal3Bundler): @@ -256,8 +436,9 @@ def calibrate_view_graph( else: num_rejected += 1 - # Step 5: Filter edges with high calibration error. - final_residuals = _fetzer_residuals(optimized_focals, edges, cam_idx_to_var_idx) + # Step 5: Filter edges with high calibration error (GLOMAP FilterImagePairs), scored at the + # APPLIED focals so edges are judged by calibrations the pipeline actually uses. + final_residuals = _fetzer_residuals(applied_focals, edges, cam_idx_to_var_idx) edges_to_remove = set() for i, (cam1, cam2, F, pp1, pp2) in enumerate(edges): edge_error = np.linalg.norm(final_residuals[2 * i : 2 * i + 2]) @@ -267,8 +448,7 @@ def calibrate_view_graph( logger.info( "View graph calibration (Fetzer): refined %d, rejected %d / %d cameras. " "Optimized focals: min=%.1f, med=%.1f, max=%.1f. " - "Filtered %d / %d edges by calibration error (threshold=%.2f). " - "Solver: %s in %d evaluations, cost %.4f → %.4f.", + "Filtered %d / %d edges by calibration error (threshold=%.2f). Solver: %s.", num_refined, num_rejected, len(sorted_cameras), @@ -278,10 +458,42 @@ def calibrate_view_graph( len(edges_to_remove), len(edges), max_edge_error, - result.message, - result.nfev, - 0.5 * np.sum(_fetzer_residuals(initial_focals, edges, cam_idx_to_var_idx) ** 2), - result.cost, + solver_info, ) return refined, edges_to_remove + + +def compute_global_view_graph_intrinsics( + v_corr_idxs_dict: Dict[Tuple[int, int], np.ndarray], + keypoints_list: List[Keypoints], + one_view_data_dict: dict, +) -> Dict[int, gtsfm_types.CALIBRATION_TYPE]: + """Estimate every camera's focal ONCE on the full verified view graph (global Fetzer). + + Unlike the per-cluster calibration -- which only sees a single cluster's edges and falls back to + the VGGT-predicted focal for cameras lacking in-cluster F-edges -- this runs over ALL verified + correspondences, giving the global estimation strength that pins focals close to GT. Initial focals + come from the loader intrinsics (the EXIF-free 1.2*maxdim heuristic) converted to Cal3Bundler; VGGT + focals are never used. Cameras with no qualifying F-edge keep the heuristic focal (still not VGGT). + + Returns Cal3Bundler intrinsics (matching the VGGT PinholeCameraCal3Bundler type used downstream), + keyed by global camera index, covering every camera in one_view_data_dict. + """ + initial_intrinsics: Dict[int, gtsfm_types.CALIBRATION_TYPE] = {} + for idx, one_view_data in one_view_data_dict.items(): + K = one_view_data.intrinsics.K() + initial_intrinsics[idx] = Cal3Bundler(float(K[0, 0]), 0.0, 0.0, float(K[0, 2]), float(K[1, 2])) + + keypoints = {idx: keypoints_list[idx] for idx in range(len(keypoints_list))} + refined, _edges_to_remove = calibrate_view_graph( + v_corr_idxs_dict=v_corr_idxs_dict, + keypoints=keypoints, + initial_intrinsics=initial_intrinsics, + ) + # NOTE: cameras Fetzer cannot refine (no qualifying F-edge, or a focal-ratio/calibration-error + # rejection) keep their 1.2*maxdim heuristic focal. A median-fill of those was TRIED and REVERTED: + # the unrefined cameras skew high-focal (narrow-FOV/telephoto, few correspondences), and for those + # the 1.2 heuristic is actually accurate (Brussels cams 12/39/68 true f/maxdim ~1.19-1.23) — filling + # them to the dataset median (~1.025) hurt 3 cameras (~14% each) to help 1 (215). Keep the heuristic. + return refined diff --git a/pipeline-viz b/pipeline-viz new file mode 100755 index 000000000..9078bd46f --- /dev/null +++ b/pipeline-viz @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# Launcher for the modern Babylon.js reconstruction viewer (alternative to ./viz). +# Sibling to ./viz — separate Flask app on port 5174. Defaults to scanning ./results. +# Usage: ./pipeline-viz [--base results] [--port 5174]. Renders static reconstructions +# (single points3D.txt + images.txt); also animates pipeline_trace/ dirs if present. +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +exec python "$SCRIPT_DIR/pipeline_visualization/server.py" "$@" diff --git a/pipeline_visualization/requirements.txt b/pipeline_visualization/requirements.txt new file mode 100644 index 000000000..c37b670ab --- /dev/null +++ b/pipeline_visualization/requirements.txt @@ -0,0 +1 @@ +Flask>=3.0 diff --git a/pipeline_visualization/server.py b/pipeline_visualization/server.py new file mode 100644 index 000000000..fb81ef965 --- /dev/null +++ b/pipeline_visualization/server.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +"""Lightweight Flask server for the GTSFM pipeline trace visualizer. + +Sibling to ./viz — completely separate code path. Scans benchmark_results/ (or +custom --base) for pipeline_trace/manifest.json files, serves the manifest + +per-stage COLMAP files to a Babylon.js frontend that animates the pipeline. + +Run: ./pipeline-viz [--base benchmark_results] [--port 5174] +""" +from __future__ import annotations + +import argparse +import json +import os +from pathlib import Path +from urllib.parse import quote + +from flask import Flask, abort, jsonify, render_template, send_file +from werkzeug.utils import safe_join + +app = Flask(__name__, static_folder="static", template_folder="templates") +app.config["SEND_FILE_MAX_AGE_DEFAULT"] = 0 + + +@app.after_request +def add_no_cache_headers(response): + response.headers["Cache-Control"] = "no-store" + return response + + +BASE_DIR = Path(os.environ.get("PIPELINE_VIZ_BASE", "results")).resolve() + + +def _run_entry(rel: str, label: str, num_stages: int, kind: str) -> dict: + return { + "id": rel, + "label": label, + "num_stages": num_stages, + "kind": kind, + "manifest_url": f"/api/runs/{quote(rel)}/manifest", + "data_url_prefix": f"/data/{quote(rel)}", + } + + +def find_runs(base_dir: Path) -> list[dict]: + """Find STATIC reconstructions and (optionally) pipeline traces under base_dir. + + - A pipeline trace is a dir with `manifest.json` AND >=1 `stage_*/` subfolder — animated. + - A static reconstruction is any dir with `points3D.txt` that isn't a trace stage — shown + as a single-stage run. This is the modern viewer's primary use here (no replay needed). + """ + runs: list[dict] = [] + seen: set[str] = set() + + # 1) Pipeline traces (if any exist under base_dir). + for manifest_path in base_dir.rglob("manifest.json"): + trace_dir = manifest_path.parent + if not any(trace_dir.glob("stage_*")): + continue + rel = trace_dir.relative_to(base_dir).as_posix() + parts = rel.split("/") + label = "/".join([p for p in parts if p != "pipeline_trace"]) or rel + try: + with manifest_path.open() as f: + num_stages = len(json.load(f).get("stages", [])) + except Exception: + num_stages = 0 + seen.add(rel) + runs.append(_run_entry(rel, label, num_stages, "trace")) + + # 2) Static reconstructions (points3D.txt). Skip trace stage_* dirs and already-listed dirs. + for pts in base_dir.rglob("points3D.txt"): + recon_dir = pts.parent + if recon_dir.name.startswith("stage_") and (recon_dir.parent / "manifest.json").exists(): + continue # a stage of a pipeline trace, not a standalone reconstruction + rel = recon_dir.relative_to(base_dir).as_posix() + if rel in seen: + continue + seen.add(rel) + runs.append(_run_entry(rel, rel, 1, "static")) + + runs.sort(key=lambda r: r["id"]) + return runs + + +@app.get("/") +def index(): + return render_template("index.html") + + +@app.get("/api/runs") +def list_runs(): + runs = find_runs(BASE_DIR) + return jsonify({"base_dir": str(BASE_DIR), "count": len(runs), "items": runs}) + + +@app.get("/api/runs//manifest") +def get_manifest(run_id): + run_dir = safe_join(str(BASE_DIR), run_id) + if run_dir is None: + abort(404) + run_path = Path(run_dir).resolve() + try: + run_path.relative_to(BASE_DIR) + except ValueError: + abort(403) + manifest_path = run_path / "manifest.json" + if manifest_path.exists(): + with manifest_path.open() as f: + return jsonify(json.load(f)) + # Static reconstruction → synthesize a single-stage manifest pointing at the dir root. + if (run_path / "points3D.txt").exists(): + return jsonify({"stages": [{"stage_name": "reconstruction", "subdir": ""}]}) + abort(404) + + +@app.get("/data/") +def serve_data(subpath): + abs_path = safe_join(str(BASE_DIR), subpath) + if abs_path is None: + abort(404) + p = Path(abs_path).resolve() + try: + p.relative_to(BASE_DIR) + except ValueError: + abort(403) + if not p.exists() or not p.is_file(): + abort(404) + return send_file(str(p), as_attachment=False) + + +def main(): + parser = argparse.ArgumentParser(description="GTSFM pipeline-viz server", add_help=False) + parser.add_argument("--base", "-b", default="results", + help="Base folder to scan for reconstructions / pipeline traces (default: results)") + parser.add_argument("--host", default="127.0.0.1", help="Host to bind.") + parser.add_argument("--port", "-p", type=int, default=5174, + help="Port (default: 5174 — sibling to ./viz on 5173).") + parser.add_argument("--help", action="help", default=argparse.SUPPRESS) + args = parser.parse_args() + + global BASE_DIR + BASE_DIR = Path(args.base).resolve() + print(f"[pipeline-viz] Serving reconstructions from: {BASE_DIR}") + if not BASE_DIR.exists(): + print(f"[pipeline-viz] WARNING: base dir does not exist; create {BASE_DIR}") + app.run(host=args.host, port=args.port, debug=False) + + +if __name__ == "__main__": + main() diff --git a/pipeline_visualization/static/style.css b/pipeline_visualization/static/style.css new file mode 100644 index 000000000..61f05093b --- /dev/null +++ b/pipeline_visualization/static/style.css @@ -0,0 +1,131 @@ +/* GTSFM Pipeline Visualizer — Snavely-2010-inspired aesthetic. + Clean white backdrop, dark wireframes (rendered in viewer.js), crisp sans-serif HUD, + minimal chrome. The point cloud and frustums carry the visual weight; the UI fades back. */ + +* { box-sizing: border-box; margin: 0; padding: 0; } + +html, body { + height: 100%; + width: 100%; + background: #f6f6f3; /* warm off-white, like a Snavely 2010 paper figure */ + font-family: -apple-system, BlinkMacSystemFont, "Inter", "Helvetica Neue", Arial, sans-serif; + color: #1a1a1a; + overflow: hidden; +} + +#renderCanvas { + position: absolute; + inset: 0; + width: 100%; + height: 100%; + display: block; + outline: none; + touch-action: none; +} + +/* Snavely-style overlay card. Positioned individually below. */ +.card { + position: absolute; + background: rgba(255, 255, 255, 0.92); + border: 1px solid rgba(0, 0, 0, 0.08); + border-radius: 6px; + padding: 12px 16px; + min-width: 200px; + font-size: 11px; + line-height: 1.6; + letter-spacing: 0.02em; + backdrop-filter: blur(4px); + box-shadow: 0 2px 8px rgba(0, 0, 0, 0.04); +} +#statsCard { top: 18px; left: 18px; } +#helpCard { top: 18px; right: 18px; min-width: 160px; opacity: 0.85; } +.card-row { + display: flex; + justify-content: space-between; + align-items: baseline; + gap: 18px; +} +.card-row .label { + text-transform: uppercase; + font-size: 9px; + font-weight: 600; + color: #888; + letter-spacing: 0.08em; +} +.card-row span:not(.label) { + font-weight: 600; + color: #1a1a1a; + font-variant-numeric: tabular-nums; +} + +/* Bottom control bar — minimal, centered, only what's needed */ +#controlBar { + position: absolute; + bottom: 24px; + left: 50%; + transform: translateX(-50%); + background: rgba(255, 255, 255, 0.95); + border: 1px solid rgba(0, 0, 0, 0.08); + border-radius: 24px; + padding: 8px 16px; + display: flex; + align-items: center; + gap: 12px; + font-size: 12px; + box-shadow: 0 2px 12px rgba(0, 0, 0, 0.06); + backdrop-filter: blur(4px); +} +#runSelect, #modeSelect { + border: none; + background: transparent; + font-family: inherit; + font-size: 12px; + color: #1a1a1a; + font-weight: 500; + outline: none; + cursor: pointer; +} +#runSelect { min-width: 200px; } +#modeSelect { min-width: 180px; } +#playBtn { + border: none; + background: #1a1a1a; + color: white; + width: 32px; + height: 32px; + border-radius: 50%; + font-size: 14px; + cursor: pointer; + display: flex; + align-items: center; + justify-content: center; +} +#playBtn:hover { background: #333; } +#timeline { + width: 320px; + cursor: pointer; + accent-color: #1a1a1a; +} +#stageIdx { + font-variant-numeric: tabular-nums; + color: #666; + min-width: 56px; + text-align: right; +} +.divider { + width: 1px; + height: 18px; + background: rgba(0,0,0,0.12); +} +.ctrl-label { + text-transform: uppercase; + font-size: 9px; + font-weight: 600; + color: #888; + letter-spacing: 0.08em; +} +#pointSize { + width: 100px; + cursor: pointer; + accent-color: #1a1a1a; +} diff --git a/pipeline_visualization/static/viewer.js b/pipeline_visualization/static/viewer.js new file mode 100644 index 000000000..67c429662 --- /dev/null +++ b/pipeline_visualization/static/viewer.js @@ -0,0 +1,772 @@ +/* GTSFM Pipeline Visualizer — minimal Babylon.js viewer. + * + * M2: discrete stage stepping (load each stage, swap point cloud). + * M3 will add: smooth interpolation between stages, wireframe frustums, + * image-RGB color sampling, transition cross-fades. + * + * Uses Babylon globals from CDN (BABYLON.*). No build step required. */ + +// ── Scene setup ───────────────────────────────────────────────────────────── +const canvas = document.getElementById("renderCanvas"); +const engine = new BABYLON.Engine(canvas, true, { preserveDrawingBuffer: true, stencil: true }); +const scene = new BABYLON.Scene(engine); +scene.clearColor = BABYLON.Color4.FromHexString("#f6f6f3ff"); // matches CSS bg + +const camera = new BABYLON.ArcRotateCamera( + "cam", -Math.PI / 2, Math.PI / 2.2, 15, + new BABYLON.Vector3(0, 0, 0), scene, +); +camera.minZ = 0.01; +camera.maxZ = 10000; +// Default Babylon up (Y-up). We compute a per-scene up-alignment rotation that +// maps the data's natural up to +Y, then apply it to all positions at parse. +// (See compute_up_alignment in scripts/visualize_gp_convergence.py for the source.) + +// Sensibilities: lower number = MORE sensitive in Babylon. +camera.angularSensibilityX = 600; // horizontal drag → alpha (yaw) +camera.angularSensibilityY = 600; // vertical drag → beta (pitch) +camera.panningSensibility = 200; // right-mouse pan +camera.wheelPrecision = 8; +camera.lowerRadiusLimit = 0.001; +camera.upperRadiusLimit = 10000; +// Prevent pole-crossing flips. JS has no Math.TWO_PI (NaN locked the camera before). +camera.lowerBetaLimit = 0.05; +camera.upperBetaLimit = Math.PI - 0.05; +// Inertia low → predictable instant response (high inertia can feel like "drift"). +camera.inertia = 0.5; +camera.panningInertia = 0.5; + +// ── Inputs: clear defaults (some have touch/twist that interferes on macOS +// trackpads), add ONLY the ones we want, THEN attachControl to wire DOM events. +// Order is critical: inputs.clear() detaches everything; addX() registers new +// inputs but doesn't bind to canvas; attachControl(canvas) binds them. +camera.inputs.clear(); +camera.inputs.addPointers(); // LMB-drag orbit, RMB-drag pan, touch +camera.inputs.addMouseWheel(); // wheel/trackpad-pinch zoom +camera.inputs.addKeyboard(); // arrow keys for continuous orbit +// Built-in arrow-key input keycodes. +camera.keysUp = [38]; // ArrowUp → beta- +camera.keysDown = [40]; // ArrowDown → beta+ +camera.keysLeft = [37]; // ArrowLeft → alpha- +camera.keysRight = [39]; // ArrowRight → alpha+ +// Attach now that inputs are fully configured. +camera.attachControl(canvas, true); + +// Make canvas focusable + auto-focus on hover so keyboard events reach it +// (without this, arrow keys go to the body and Babylon never sees them). +canvas.tabIndex = 0; +canvas.style.outline = "none"; // hide focus ring +canvas.addEventListener("mouseenter", () => canvas.focus()); + +// ── Up-alignment + scene framing (ported from scripts/visualize_gp_convergence.py) +// Active scene transform: 3x3 rotation that maps data-frame → Babylon-frame (Y-up). +// Computed once per RUN (not per stage) so all stages share the same orientation. +let sceneRotation = null; // 3x3 row-major; null → identity +let sceneViewCenter = null; // [x,y,z] median of all positions in transformed frame +let sceneViewRange = null; // 90th percentile distance from center in transformed frame + +function applySceneRotation(x, y, z) { + if (!sceneRotation) return [x, y, z]; + const R = sceneRotation; + return [ + R[0][0]*x + R[0][1]*y + R[0][2]*z, + R[1][0]*x + R[1][1]*y + R[1][2]*z, + R[2][0]*x + R[2][1]*y + R[2][2]*z, + ]; +} + +// Compute a rotation that aligns scene "up" with Babylon's +Y. +// Strategy mirrors compute_up_alignment in the matplotlib script: +// 1) Avg camera image-up in world (wRi @ [0,-1,0]); use if cameras agree (|avg|>0.5) +// 2) Else PCA least-variance axis over all positions +// 3) Flip if landmarks sit BELOW cameras under the proposed up +function computeUpAlignment(camPositions, camRotations, landmarks) { + let sceneUp = null; + + if (camRotations && camRotations.length > 0) { + // R from parseImages is cam_from_world (COLMAP convention). The world-space + // "image up" vector is wRi @ [0,-1,0] = (cam_from_world)^T @ [0,-1,0] + // = -row1(cam_from_world) = (-R[1][0], -R[1][1], -R[1][2]). + // (Earlier this code used -col1(R) which gave wrong direction → upside-down scene.) + let sx = 0, sy = 0, sz = 0; + for (const R of camRotations) { + sx += -R[1][0]; sy += -R[1][1]; sz += -R[1][2]; + } + const n = camRotations.length; + sx /= n; sy /= n; sz /= n; + const mag = Math.hypot(sx, sy, sz); + if (mag > 0.5) sceneUp = [sx / mag, sy / mag, sz / mag]; + } + + if (!sceneUp) { + // PCA least-variance axis as fallback. Use power iteration on the + // smallest eigenvalue of the covariance matrix (cheap, no SVD library). + const all = landmarks && landmarks.length > 0 ? camPositions.concat(landmarks) : camPositions; + if (all.length < 3) return null; + let mx = 0, my = 0, mz = 0; + for (const [x, y, z] of all) { mx += x; my += y; mz += z; } + mx /= all.length; my /= all.length; mz /= all.length; + let cxx = 0, cyy = 0, czz = 0, cxy = 0, cxz = 0, cyz = 0; + for (const [x, y, z] of all) { + const dx = x - mx, dy = y - my, dz = z - mz; + cxx += dx*dx; cyy += dy*dy; czz += dz*dz; + cxy += dx*dy; cxz += dx*dz; cyz += dy*dz; + } + // Smallest eigenvector via inverse power iteration on (C - μI)⁻¹. + // For our purposes: pick the world axis with smallest variance as a hint. + if (czz <= cxx && czz <= cyy) sceneUp = [0, 0, 1]; + else if (cyy <= cxx) sceneUp = [0, 1, 0]; + else sceneUp = [1, 0, 0]; + } + + // Heuristic: landmarks should sit above cameras under the proposed up. + if (landmarks && landmarks.length > 0 && camPositions.length > 0) { + let lcx=0, lcy=0, lcz=0; for (const [x,y,z] of landmarks) {lcx+=x; lcy+=y; lcz+=z;} + lcx /= landmarks.length; lcy /= landmarks.length; lcz /= landmarks.length; + let ccx=0, ccy=0, ccz=0; for (const [x,y,z] of camPositions) {ccx+=x; ccy+=y; ccz+=z;} + ccx /= camPositions.length; ccy /= camPositions.length; ccz /= camPositions.length; + const dot = (lcx-ccx)*sceneUp[0] + (lcy-ccy)*sceneUp[1] + (lcz-ccz)*sceneUp[2]; + if (dot < 0) sceneUp = sceneUp.map(v => -v); + } + + // Build rotation R mapping sceneUp → +Y. Pick a stable reference axis, + // then construct an orthonormal basis [right, up, fwd] with up = sceneUp. + let ref = [1, 0, 0]; + if (Math.abs(ref[0]*sceneUp[0] + ref[1]*sceneUp[1] + ref[2]*sceneUp[2]) > 0.95) ref = [0, 1, 0]; + // x_axis = sceneUp × ref (orthogonal to up) + const xa = [ + sceneUp[1]*ref[2] - sceneUp[2]*ref[1], + sceneUp[2]*ref[0] - sceneUp[0]*ref[2], + sceneUp[0]*ref[1] - sceneUp[1]*ref[0], + ]; + const xn = Math.hypot(...xa); for (let i = 0; i < 3; i++) xa[i] /= (xn + 1e-12); + // z_axis = sceneUp × xa + const za = [ + sceneUp[1]*xa[2] - sceneUp[2]*xa[1], + sceneUp[2]*xa[0] - sceneUp[0]*xa[2], + sceneUp[0]*xa[1] - sceneUp[1]*xa[0], + ]; + // R rows: [x_axis, sceneUp, z_axis] — input p maps to (xa·p, sceneUp·p, za·p), + // so the second component (Y) becomes sceneUp · p ⇒ +Y is the data's up. + return [ + [xa[0], xa[1], xa[2]], + [sceneUp[0], sceneUp[1], sceneUp[2]], + [za[0], za[1], za[2]], + ]; +} + +// View center = median of positions in transformed frame; range = 90th percentile +// distance from center × 1.05. Robust to outlier landmarks that would otherwise +// blow up a bounding-box fit. +function computeViewSetup(transformedPoints) { + if (transformedPoints.length === 0) return null; + // Per-axis median. + const xs = transformedPoints.map(p => p[0]).sort((a, b) => a - b); + const ys = transformedPoints.map(p => p[1]).sort((a, b) => a - b); + const zs = transformedPoints.map(p => p[2]).sort((a, b) => a - b); + const mid = arr => arr[Math.floor(arr.length / 2)]; + const center = [mid(xs), mid(ys), mid(zs)]; + // 90th-percentile distance from center. + const dists = transformedPoints.map(p => Math.hypot(p[0]-center[0], p[1]-center[1], p[2]-center[2])).sort((a, b) => a - b); + const range = dists[Math.floor(dists.length * 0.9)] * 1.05; + return { center, range }; +} + +// Initial view azimuth: place camera on the SAME XZ-plane side as the camera +// cluster (i.e. where the photographers actually stood). This is more robust than +// the PCA-perpendicular approach — PCA returns one of two normals arbitrarily, +// which can land you behind/inside the structure on some scenes. +function computeAutoCameraAngles(transformedPoints, transformedCamPositions) { + if (!transformedCamPositions || transformedCamPositions.length === 0 || transformedPoints.length < 3) { + return { alpha: -Math.PI / 2, beta: Math.PI / 3 }; + } + // Centroid of all points (≈ structure center) and centroid of cameras (≈ photographer side). + let pmx = 0, pmz = 0; + for (const [x, , z] of transformedPoints) { pmx += x; pmz += z; } + pmx /= transformedPoints.length; pmz /= transformedPoints.length; + let cmx = 0, cmz = 0; + for (const [x, , z] of transformedCamPositions) { cmx += x; cmz += z; } + cmx /= transformedCamPositions.length; cmz /= transformedCamPositions.length; + // Direction from structure center to photographer cluster, in XZ plane. + // Camera should sit on this side looking back at the structure. + const vx = cmx - pmx, vz = cmz - pmz; + const norm = Math.hypot(vx, vz); + // Babylon ArcRotateCamera: alpha = atan2(z, x), camera position is at + // (target.x + r*cos(α)*sin(β), target.y + r*cos(β), target.z + r*sin(α)*sin(β)). + // We want camera toward (vx, vz) from target, so alpha = atan2(vz, vx). + let alpha = norm > 1e-6 ? Math.atan2(vz, vx) : -Math.PI / 2; + // Elev 30° above horizontal: Babylon beta from +Y axis → π/3. + const beta = Math.PI / 3; + return { alpha, beta }; +} + +new BABYLON.HemisphericLight("h", new BABYLON.Vector3(0, 1, 0), scene).intensity = 0.85; + +engine.runRenderLoop(() => scene.render()); +window.addEventListener("resize", () => engine.resize()); + +// Camera-flip / reset hotkeys. Arrow keys are handled by Babylon's built-in +// addKeyboard input. +// R → reset to initial auto-framing +// F → flip front/back (alpha += π) +// T → flip top/bottom by inverting the scene-up axis (rebuilds rotation, +// re-applies it to the data, and re-frames). This is a true vertical +// flip — handy if up-alignment landed inverted. +let savedView = null; +function flipSceneUpAndReframe() { + if (!sceneRotation || !currentRun) return; + // Negate the second row of sceneRotation, which is the row that maps data → +Y. + // This effectively flips the up axis. + sceneRotation[1] = sceneRotation[1].map(v => -v); + // Reload current stage so re-parsing uses the updated rotation. + loadStage(currentStageIdx, { animate: false }); + // Re-frame: target.y inverts, so reset camera to recomputed setup. + // For simplicity, just press R after to re-snap to a clean initial. + if (savedView) { + savedView.center[1] = -savedView.center[1]; + camera.target.set(savedView.center[0], savedView.center[1], savedView.center[2]); + } +} +let uiHidden = false; +window.addEventListener("keydown", (e) => { + if (e.target.tagName === "SELECT" || e.target.tagName === "INPUT") return; + if (e.key === "r" || e.key === "R") { + if (savedView) { + camera.target.set(savedView.center[0], savedView.center[1], savedView.center[2]); + camera.alpha = savedView.alpha; + camera.beta = savedView.beta; + camera.radius = savedView.radius; + } + e.preventDefault(); + } else if (e.key === "f" || e.key === "F") { + camera.alpha += Math.PI; // front ↔ back + e.preventDefault(); + } else if (e.key === "t" || e.key === "T") { + flipSceneUpAndReframe(); // top ↔ bottom (true vertical flip via scene-up flip) + e.preventDefault(); + } else if (e.key === "h" || e.key === "H") { + // Hide ALL overlays for a clean money shot; press again to restore. + uiHidden = !uiHidden; + for (const id of ["statsCard", "helpCard", "controlBar"]) { + const el = document.getElementById(id); + if (el) el.style.display = uiHidden ? "none" : ""; + } + e.preventDefault(); + } +}); + +// ── State ─────────────────────────────────────────────────────────────────── +let currentRun = null; // { id, manifest_url, data_url_prefix, manifest } +let currentStageIdx = 0; +let pointsMesh = null; +let frustumLines = null; // single LinesSystem mesh for all frustums +let isPlaying = false; +let playTimer = null; +let prevStageData = null; // previous stage points (for points LERP) +let prevStageImages = null; // previous stage cameras (for frustum LERP) +let lerpAnimRAF = null; // requestAnimationFrame handle for points lerp +let frustumLerpRAF = null; // requestAnimationFrame handle for frustum lerp +const LERP_MS = 600; // ms to interpolate between consecutive stages +let pointSize = 2.0; // current point cloud size, controlled by UI slider + +// ── COLMAP parsing (minimal — just what we need) ──────────────────────────── +async function fetchText(url) { + const r = await fetch(url); + if (!r.ok) throw new Error(`fetch ${url} failed: ${r.status}`); + return await r.text(); +} + +function parsePoints3D(text) { + // Format per line: POINT3D_ID X Y Z R G B ERROR TRACK[] + // Applies sceneRotation if set so all positions live in the aligned (Y-up) frame. + const xyz = []; + const rgb = []; + for (const raw of text.split("\n")) { + const line = raw.trim(); + if (!line || line.startsWith("#")) continue; + const parts = line.split(/\s+/); + if (parts.length < 7) continue; + const [bx, by, bz] = applySceneRotation( + parseFloat(parts[1]), parseFloat(parts[2]), parseFloat(parts[3]) + ); + xyz.push(bx, by, bz); + rgb.push(parseInt(parts[4]) / 255, parseInt(parts[5]) / 255, parseInt(parts[6]) / 255); + } + return { xyz, rgb, count: xyz.length / 3 }; +} + +// Parse images.txt: each image is 2 lines (pose + 2D points). Returns {pose, name}[]. +// COLMAP stores cam_from_world. We invert to world-from-cam for rendering. +function parseImages(text) { + const images = []; + const lines = text.split("\n").filter(l => l.trim() && !l.trim().startsWith("#")); + for (let i = 0; i < lines.length; i += 2) { + const parts = lines[i].trim().split(/\s+/); + if (parts.length < 9) continue; + const qw = parseFloat(parts[1]), qx = parseFloat(parts[2]), + qy = parseFloat(parts[3]), qz = parseFloat(parts[4]); + const tx = parseFloat(parts[5]), ty = parseFloat(parts[6]), tz = parseFloat(parts[7]); + const name = parts.slice(9).join(" "); + // World-from-cam: world_pos = -R^T @ t, where R is from quaternion. + // R from (qw, qx, qy, qz): + const xx = qx * qx, yy = qy * qy, zz = qz * qz; + const xy = qx * qy, xz = qx * qz, yz = qy * qz; + const wx = qw * qx, wy = qw * qy, wz = qw * qz; + const R = [ + [1 - 2*(yy + zz), 2*(xy - wz), 2*(xz + wy)], + [2*(xy + wz), 1 - 2*(xx + zz), 2*(yz - wx)], + [2*(xz - wy), 2*(yz + wx), 1 - 2*(xx + yy)], + ]; + // R^T @ t = column-by-column dot product (camera center in DATA coords) + const cxC = -(R[0][0]*tx + R[1][0]*ty + R[2][0]*tz); + const cyC = -(R[0][1]*tx + R[1][1]*ty + R[2][1]*tz); + const czC = -(R[0][2]*tx + R[1][2]*ty + R[2][2]*tz); + // Apply scene rotation so center is in viewer (Y-up aligned) coords. + // Keep the camera rotation matrix R in DATA frame; renderFrustums will + // rotate corner offsets through R then through sceneRotation. + const [cx, cy, cz] = applySceneRotation(cxC, cyC, czC); + images.push({ position: [cx, cy, cz], R: R, name: name, dataPos: [cxC, cyC, czC] }); + } + return images; +} + +// ── Scene rendering ───────────────────────────────────────────────────────── +function disposePoints() { + if (pointsMesh) { pointsMesh.dispose(); pointsMesh = null; } +} +function disposeFrustums() { + if (frustumLines) { frustumLines.dispose(); frustumLines = null; } +} + +function renderStagePoints({ xyz, rgb, count }) { + disposePoints(); + if (count === 0) return; + + const positions = new Float32Array(xyz); + const colors = new Float32Array(count * 4); + for (let i = 0; i < count; i++) { + colors[i * 4 + 0] = rgb[i * 3 + 0]; + colors[i * 4 + 1] = rgb[i * 3 + 1]; + colors[i * 4 + 2] = rgb[i * 3 + 2]; + colors[i * 4 + 3] = 1.0; + } + const mesh = new BABYLON.Mesh("pts", scene); + const vd = new BABYLON.VertexData(); + vd.positions = positions; + vd.colors = colors; + vd.applyToMesh(mesh, true); // updatable=true so we can morph for LERP + const mat = new BABYLON.StandardMaterial("pts_mat", scene); + mat.disableLighting = true; + mat.emissiveColor = new BABYLON.Color3(1, 1, 1); + mat.pointsCloud = true; + mat.pointSize = pointSize; + mesh.material = mat; + pointsMesh = mesh; +} + +// Render dark wireframe frustums per camera. Snavely-2010 look: thin dark lines, +// uniform size, no shading. Frustum = 4 apex→corner lines + 4 corner-quad lines. +// Build the array of line segments for all frustums. Pure compute, no Babylon side effects. +function buildFrustumLines(images, viewRange) { + // Frustum size = view_range * 0.03 (same constant as the matplotlib script). + const s = Math.max(0.001, viewRange * 0.03); + const half = s * 0.5; + const localCorners = [ + [-half, -half, s], + [ half, -half, s], + [ half, half, s], + [-half, half, s], + ]; + const lines = []; + for (const img of images) { + const apex = new BABYLON.Vector3(img.position[0], img.position[1], img.position[2]); + const worldCorners = localCorners.map(([lx, ly, lz]) => { + // R^T @ local in DATA frame, then apply sceneRotation. + const wxC = img.R[0][0]*lx + img.R[1][0]*ly + img.R[2][0]*lz; + const wyC = img.R[0][1]*lx + img.R[1][1]*ly + img.R[2][1]*lz; + const wzC = img.R[0][2]*lx + img.R[1][2]*ly + img.R[2][2]*lz; + const [bx, by, bz] = applySceneRotation(wxC, wyC, wzC); + return new BABYLON.Vector3(apex.x + bx, apex.y + by, apex.z + bz); + }); + for (const c of worldCorners) lines.push([apex, c]); + for (let i = 0; i < 4; i++) lines.push([worldCorners[i], worldCorners[(i + 1) % 4]]); + } + return lines; +} + +function renderFrustums(images, viewRange) { + disposeFrustums(); + if (!images.length) return; + const lines = buildFrustumLines(images, viewRange); + // updatable=true lets us re-feed lines via `instance:` for in-place vertex updates. + frustumLines = BABYLON.MeshBuilder.CreateLineSystem("frustums", { + lines: lines, + updatable: true, + }, scene); + frustumLines.color = new BABYLON.Color3(0.85, 0.15, 0.15); // muted red, like matplotlib + frustumLines.alpha = 0.85; +} + +// Update existing frustum mesh in place (no dispose/recreate). Babylon updates +// vertex positions of the same buffer when `instance:` is passed. +function updateFrustumsInPlace(images, viewRange) { + if (!frustumLines) { + renderFrustums(images, viewRange); + return; + } + const lines = buildFrustumLines(images, viewRange); + frustumLines = BABYLON.MeshBuilder.CreateLineSystem("frustums", { + lines: lines, + instance: frustumLines, + }, scene); +} + +// Smooth LERP between two stages' camera frusta. Component-wise lerp for both +// position + rotation matrix (not a true slerp, but visually fine for small +// per-stage refinements). Falls back to discrete swap if camera count differs. +function lerpFrustums(prevImgs, nextImgs, viewRange, durationMs) { + if (frustumLerpRAF) cancelAnimationFrame(frustumLerpRAF); + if (!prevImgs || !frustumLines || prevImgs.length !== nextImgs.length) { + renderFrustums(nextImgs, viewRange); + return; + } + const t0 = performance.now(); + const tick = () => { + const elapsed = performance.now() - t0; + const u = Math.min(1, elapsed / durationMs); + const e = u * u * (3 - 2 * u); // smoothstep + const oneMinusE = 1 - e; + // Build interpolated images (positions + rotations). + const interp = nextImgs.map((nx, i) => { + const pv = prevImgs[i]; + return { + position: [ + pv.position[0] * oneMinusE + nx.position[0] * e, + pv.position[1] * oneMinusE + nx.position[1] * e, + pv.position[2] * oneMinusE + nx.position[2] * e, + ], + R: [ + [pv.R[0][0]*oneMinusE+nx.R[0][0]*e, pv.R[0][1]*oneMinusE+nx.R[0][1]*e, pv.R[0][2]*oneMinusE+nx.R[0][2]*e], + [pv.R[1][0]*oneMinusE+nx.R[1][0]*e, pv.R[1][1]*oneMinusE+nx.R[1][1]*e, pv.R[1][2]*oneMinusE+nx.R[1][2]*e], + [pv.R[2][0]*oneMinusE+nx.R[2][0]*e, pv.R[2][1]*oneMinusE+nx.R[2][1]*e, pv.R[2][2]*oneMinusE+nx.R[2][2]*e], + ], + }; + }); + updateFrustumsInPlace(interp, viewRange); + if (u < 1) { + frustumLerpRAF = requestAnimationFrame(tick); + } else { + frustumLerpRAF = null; + } + }; + frustumLerpRAF = requestAnimationFrame(tick); +} + +// Paintball animation — for retri/densify stages. Each new point fires from a +// random camera frustum out into the scene, with per-point delay so they pepper +// in over time (not all at once). Phase 1: previous mesh fades out; Phase 2: +// fresh mesh built at camera positions, points fly to their final positions. +// Looks like a paintball gun spraying paint into the scene. +function paintballStagePoints(prevImages, next, durationMs) { + if (lerpAnimRAF) cancelAnimationFrame(lerpAnimRAF); + if (!next.count || !prevImages || prevImages.length === 0) { + renderStagePoints(next); + return; + } + // Pick a random source camera per point. + const camPositions = prevImages.map(img => img.position); + const nCams = camPositions.length; + const startPos = new Float32Array(next.count * 3); + for (let i = 0; i < next.count; i++) { + const c = camPositions[Math.floor(Math.random() * nCams)]; + startPos[i * 3] = c[0]; + startPos[i * 3 + 1] = c[1]; + startPos[i * 3 + 2] = c[2]; + } + // Per-point fire delay (0-50% of duration) → staggered, paintball spray feel. + const delays = new Float32Array(next.count); + for (let i = 0; i < next.count; i++) delays[i] = Math.random() * 0.5; + + // Build the new mesh AT the start (camera) positions, with final colors. + renderStagePoints({ xyz: Array.from(startPos), rgb: next.rgb, count: next.count }); + const endPos = new Float32Array(next.xyz); + + // Per-point alpha: hidden until the delay fires, then 1.0 (so points pop in + // rather than streaking from the camera since frame 0). Babylon points-cloud + // material doesn't support per-vertex alpha cheaply, so we just animate + // position from camera → final and let perspective handle the "shoot" motion. + const t0 = performance.now(); + const tick = () => { + if (!pointsMesh) { lerpAnimRAF = null; return; } + const elapsed = performance.now() - t0; + const u = Math.min(1, elapsed / durationMs); + const pos = pointsMesh.getVerticesData(BABYLON.VertexBuffer.PositionKind); + for (let i = 0; i < next.count; i++) { + // Each point's local progress: starts at delay, finishes at 1. + const localU = Math.max(0, Math.min(1, (u - delays[i]) / (1 - delays[i] + 1e-9))); + // Exponential ease-out — fast initial flight, slows on approach (paintball arc). + const e = 1 - Math.pow(1 - localU, 3); + const ix = i * 3; + pos[ix] = startPos[ix] * (1 - e) + endPos[ix] * e; + pos[ix + 1] = startPos[ix + 1] * (1 - e) + endPos[ix + 1] * e; + pos[ix + 2] = startPos[ix + 2] * (1 - e) + endPos[ix + 2] * e; + } + pointsMesh.updateVerticesData(BABYLON.VertexBuffer.PositionKind, pos); + if (u < 1) { + lerpAnimRAF = requestAnimationFrame(tick); + } else { + lerpAnimRAF = null; + } + }; + lerpAnimRAF = requestAnimationFrame(tick); +} + +// Smooth LERP between two point clouds. If counts differ, just discrete-swap +// (can't morph different vertex counts cleanly without correspondence). +function lerpStagePoints(prev, next, durationMs) { + if (lerpAnimRAF) cancelAnimationFrame(lerpAnimRAF); + if (!prev || !pointsMesh || prev.count !== next.count) { + renderStagePoints(next); + return; + } + const positions = pointsMesh.getVerticesData(BABYLON.VertexBuffer.PositionKind); + const start = new Float32Array(positions); // current shown positions + const end = new Float32Array(next.xyz); + const t0 = performance.now(); + const tick = () => { + const elapsed = performance.now() - t0; + const u = Math.min(1, elapsed / durationMs); + const e = u * u * (3 - 2 * u); // smoothstep ease + for (let i = 0; i < positions.length; i++) { + positions[i] = start[i] * (1 - e) + end[i] * e; + } + pointsMesh.updateVerticesData(BABYLON.VertexBuffer.PositionKind, positions); + if (u < 1) { + lerpAnimRAF = requestAnimationFrame(tick); + } else { + lerpAnimRAF = null; + // Swap colors at the end (final stage's RGB). + const colors = new Float32Array(next.count * 4); + for (let i = 0; i < next.count; i++) { + colors[i * 4 + 0] = next.rgb[i * 3 + 0]; + colors[i * 4 + 1] = next.rgb[i * 3 + 1]; + colors[i * 4 + 2] = next.rgb[i * 3 + 2]; + colors[i * 4 + 3] = 1.0; + } + pointsMesh.updateVerticesData(BABYLON.VertexBuffer.ColorKind, colors); + } + }; + lerpAnimRAF = requestAnimationFrame(tick); +} + +// Apply view setup computed via compute_up_alignment + compute_view_setup. +// Called once per RUN (not per stage) — sceneRotation, sceneViewCenter and +// sceneViewRange are scene-wide invariants. +function applyViewSetup(transformedPoints, transformedCamPositions) { + const setup = computeViewSetup(transformedPoints); + if (!setup) return; + sceneViewCenter = setup.center; + sceneViewRange = setup.range; + const angles = computeAutoCameraAngles(transformedPoints, transformedCamPositions); + camera.target = new BABYLON.Vector3(setup.center[0], setup.center[1], setup.center[2]); + // Radius ≈ 2.5 × view_range so the 90th-pct sphere sits comfortably in frame. + camera.radius = setup.range * 2.5; + camera.alpha = angles.alpha; + camera.beta = angles.beta; + // Snapshot the view for "R" reset key. + savedView = { center: setup.center, alpha: angles.alpha, beta: angles.beta, radius: setup.range * 2.5 }; +} + +// ── Stage loading ─────────────────────────────────────────────────────────── +async function loadStage(idx, { animate = true } = {}) { + if (!currentRun) return; + const stage = currentRun.manifest.stages[idx]; + if (!stage) return; + const stageUrl = stage.subdir + ? `${currentRun.data_url_prefix}/${encodeURIComponent(stage.subdir)}` + : currentRun.data_url_prefix; // static reconstruction: files live at the run-dir root + const [pointsText, imagesText] = await Promise.all([ + fetchText(`${stageUrl}/points3D.txt`), + fetchText(`${stageUrl}/images.txt`), + ]); + const points = parsePoints3D(pointsText); + const images = parseImages(imagesText); + + // Paintball effect ONLY for retri — that's where the structure dramatically + // rebuilds (multi-view re-triangulation creates a fresh point set from the + // correspondence graph). Densify is a quieter additive step where the existing + // structure stays stable and we just lerp in 2-view supplements. + const isPaintball = stage.stage_name === "retri"; + if (idx === 0 || !animate || !pointsMesh) { + renderStagePoints(points); + } else if (isPaintball && prevStageImages) { + paintballStagePoints(prevStageImages, points, 1800); + } else { + lerpStagePoints(prevStageData, points, LERP_MS); + } + // Frustums: smooth LERP same as points when camera count matches. Otherwise + // (rare; camera count is stable across stages in our pipeline) discrete swap. + if (idx === 0 || !animate || !frustumLines) { + renderFrustums(images, sceneViewRange ?? 1.0); + } else { + lerpFrustums(prevStageImages, images, sceneViewRange ?? 1.0, LERP_MS); + } + prevStageData = points; + prevStageImages = images; + + // Update HUD + document.getElementById("stageLabel").textContent = + `${stage.stage_name} (${idx + 1}/${currentRun.manifest.stages.length})`; + document.getElementById("numCameras").textContent = images.length.toLocaleString(); + document.getElementById("numPoints").textContent = points.count.toLocaleString(); + document.getElementById("stageIdx").textContent = + `${idx + 1} / ${currentRun.manifest.stages.length}`; + document.getElementById("timeline").value = idx; + currentStageIdx = idx; +} + +// Trim a manifest to the camera-ready set: first 10 GP iters + final, first 5 +// of each per-iter BA group + final, plus all singleton stages (ba_input, +// outer_ba_iter_*, retri, retri_final_ba, per_obs_filter, min_tri_filter, densify). +function trimManifest(manifest, gpHead = 10, baHead = 5) { + // Group stages by their iter-prefix (everything before "_iter_NNN"). + const groups = {}; // prefix → ordered array of {idx, stage} + manifest.stages.forEach((s, idx) => { + const name = s.stage_name; + let prefix = null; + if (name.startsWith("gp_iter_")) prefix = "gp"; + else if (name.includes("_iter_")) prefix = name.replace(/_iter_\d+$/, ""); + if (prefix) { + (groups[prefix] = groups[prefix] || []).push({ idx, stage: s }); + } + }); + const keep = new Set(); + for (const [prefix, entries] of Object.entries(groups)) { + const headN = prefix === "gp" ? gpHead : baHead; + for (let i = 0; i < Math.min(headN, entries.length); i++) keep.add(entries[i].idx); + keep.add(entries[entries.length - 1].idx); // always include last + } + const trimmed = manifest.stages.filter((s, idx) => { + const name = s.stage_name; + const isIter = name.startsWith("gp_iter_") || name.includes("_iter_"); + return !isIter || keep.has(idx); + }); + return { ...manifest, stages: trimmed }; +} + +async function selectRun(runId) { + const all = await (await fetch("/api/runs")).json(); + const item = all.items.find(r => r.id === runId); + if (!item) return; + let manifest = await (await fetch(item.manifest_url)).json(); + const mode = document.getElementById("modeSelect").value; + if (mode === "trim") { + const before = manifest.stages.length; + manifest = trimManifest(manifest); + console.log(`[trim] ${before} → ${manifest.stages.length} stages`); + } + currentRun = { ...item, manifest }; + document.getElementById("runLabel").textContent = item.label; + const tl = document.getElementById("timeline"); + tl.max = String(Math.max(0, manifest.stages.length - 1)); + tl.value = "0"; + + // Static reconstruction (single stage) → hide the trace-only controls (mode / play / timeline / stage). + const isStatic = manifest.stages.length <= 1; + for (const el of document.querySelectorAll('[data-role="trace-only"]')) { + el.style.display = isStatic ? "none" : ""; + } + + // ── Compute scene-wide up-alignment + view setup from a REPRESENTATIVE stage. + // First GP stages are noisy (random init) — pick the LAST stage (most converged + // structure) so the up-alignment uses meaningful camera rotations + landmarks. + // Reset rotation first so this stage's parse is in raw data coords. + sceneRotation = null; + sceneViewCenter = null; + sceneViewRange = null; + prevStageData = null; + prevStageImages = null; + disposePoints(); disposeFrustums(); + const lastStage = manifest.stages[manifest.stages.length - 1]; + const lastUrl = lastStage.subdir + ? `${item.data_url_prefix}/${encodeURIComponent(lastStage.subdir)}` + : item.data_url_prefix; + const [refPointsText, refImagesText] = await Promise.all([ + fetchText(`${lastUrl}/points3D.txt`), + fetchText(`${lastUrl}/images.txt`), + ]); + const refPoints = parsePoints3D(refPointsText); // identity rotation right now + const refImages = parseImages(refImagesText); + const camPositions = refImages.map(img => img.dataPos); + const camRotations = refImages.map(img => img.R); + const landmarkArr = []; + for (let i = 0; i < refPoints.count; i++) { + landmarkArr.push([refPoints.xyz[i*3], refPoints.xyz[i*3+1], refPoints.xyz[i*3+2]]); + } + sceneRotation = computeUpAlignment(camPositions, camRotations, landmarkArr); + // Apply the rotation to landmarks AND camera positions for view setup. + const transformedPoints = landmarkArr.map(([x, y, z]) => applySceneRotation(x, y, z)); + const transformedCamPositions = camPositions.map(([x, y, z]) => applySceneRotation(x, y, z)); + applyViewSetup(transformedPoints, transformedCamPositions); + + // Now load stage 0 fresh — parsers will apply sceneRotation automatically. + await loadStage(0); +} + +// ── UI wiring ─────────────────────────────────────────────────────────────── +async function init() { + const data = await (await fetch("/api/runs")).json(); + const sel = document.getElementById("runSelect"); + for (const run of data.items) { + const opt = document.createElement("option"); + opt.value = run.id; + opt.textContent = `${run.label} (${run.num_stages} stages)`; + sel.appendChild(opt); + } + if (data.items.length > 0) { + sel.value = data.items[0].id; + await selectRun(data.items[0].id); + } else { + document.getElementById("runLabel").textContent = "(no runs found)"; + } + sel.addEventListener("change", () => selectRun(sel.value)); + + // Trim-mode toggle: re-fetch + re-filter the manifest, reset to stage 0. + document.getElementById("modeSelect").addEventListener("change", () => + selectRun(sel.value)); + + document.getElementById("timeline").addEventListener("input", e => + loadStage(parseInt(e.target.value))); + + // Point-size slider — live update without rebuilding the mesh. Defensive + // null-check for stale-cached HTML / pre-refresh state. + const sizeSlider = document.getElementById("pointSize"); + if (sizeSlider) { + pointSize = parseFloat(sizeSlider.value) || pointSize; + sizeSlider.addEventListener("input", e => { + pointSize = parseFloat(e.target.value); + if (pointsMesh && pointsMesh.material) { + pointsMesh.material.pointSize = pointSize; + } + }); + } + + const playBtn = document.getElementById("playBtn"); + playBtn.addEventListener("click", () => { + isPlaying = !isPlaying; + playBtn.textContent = isPlaying ? "❚❚" : "▶"; + if (isPlaying) { + playTimer = setInterval(() => { + const next = (currentStageIdx + 1) % currentRun.manifest.stages.length; + loadStage(next); // smooth LERP runs in the background while we wait + }, LERP_MS + 200); // step a hair after the LERP finishes + } else { + clearInterval(playTimer); + } + }); +} + +init().catch(err => { + console.error(err); + document.getElementById("runLabel").textContent = "ERROR — check server logs"; +}); diff --git a/pyproject.toml b/pyproject.toml index 401169758..0919a0b5b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -54,6 +54,7 @@ dependencies = [ # 3rd party algorithms for different modules "kornia>=0.7.3", "pycolmap==3.11.1", + "poselib>=2.0", # PoseLibVerifier (two-view relative-pose, peak frontend) # IO "h5py", diff --git a/scripts/build_colmap_db.sh b/scripts/build_colmap_db.sh new file mode 100644 index 000000000..7b93a50a3 --- /dev/null +++ b/scripts/build_colmap_db.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# Build a COLMAP frontend database.db (keypoints + geometrically-verified two_view_geometries) for a +# dataset, to feed the GTSFM colmapdb configs (vggt[_omega]_sift_frontend_megaloc_phototourism_colmapdb). +# This replaces GTSFM's Dask SIFT frontend, which OOMs on large (1000+-image) scenes — COLMAP's native +# CUDA SIFT + vocab-tree matching is far faster and memory-efficient. Run on a GPU node. +# +# Usage: +# bash scripts/build_colmap_db.sh [IMAGE_SUBDIR=images] [MAX_FEATURES=8192] +# Example: +# bash scripts/build_colmap_db.sh /storage/scratch1/5//.../benchmarks/st_peters_square +# +# Produces /database.db (+ a vocab tree alongside it). Then run, e.g.: +# ./run --config_name vggt_omega_sift_frontend_megaloc_phototourism_colmapdb \ +# loader._target_=gtsfm.loader.Colmap loader.dataset_dir= \ +# +loader.colmap_files_subdir=sparse +loader.use_gt_intrinsics=false +set -euo pipefail + +DATASET_DIR="${1:?usage: build_colmap_db.sh [IMAGE_SUBDIR] [MAX_FEATURES]}" +IMAGE_SUBDIR="${2:-images}" +MAX_FEATURES="${3:-8192}" + +# Resolve any ~/scratch symlink to the real /storage path so the COLMAP apptainer container can see it. +DATASET_DIR="$(cd "$DATASET_DIR" && pwd -P)" +IMG="$DATASET_DIR/$IMAGE_SUBDIR" +DB="$DATASET_DIR/database.db" +VOCAB="$DATASET_DIR/vocab_tree_flickr100K_words256K.bin" +# Bind the data dir into the (containerized) COLMAP so it can read images + write the db. +export APPTAINER_BIND="${APPTAINER_BIND:+$APPTAINER_BIND,}$DATASET_DIR" + +[ -d "$IMG" ] || { echo "ERROR: image dir not found: $IMG"; exit 1; } + +# On PACE, `colmap` is provided as a module-defined apptainer wrapper. Make it available even when this +# script runs as a (non-interactive) subprocess: init Lmod, then load the module. +if ! command -v colmap >/dev/null 2>&1; then + if ! command -v module >/dev/null 2>&1; then + for f in /etc/profile.d/z00_lmod.sh /etc/profile.d/lmod.sh /usr/share/lmod/lmod/init/bash; do + [ -f "$f" ] && source "$f" && break + done + fi + module load colmap 2>/dev/null || true +fi +command -v colmap >/dev/null 2>&1 || { + echo "ERROR: 'colmap' is not available. Run 'module load colmap' first (and 'export -f colmap' if invoking" + echo " this as 'bash …'), or 'source scripts/build_colmap_db.sh …' so the colmap function is in scope." + exit 1 +} + +echo "📂 dataset : $DATASET_DIR" +echo "🖼️ images : $IMG ($(ls "$IMG" | wc -l) files)" +echo "🗄️ database: $DB" + +# 1) Feature extraction — try GPU SIFT, fall back to CPU if it can't get an OpenGL context (headless node). +echo "🔍 feature_extractor (GPU SIFT, max_num_features=$MAX_FEATURES)…" +if ! colmap feature_extractor --database_path "$DB" --image_path "$IMG" \ + --SiftExtraction.use_gpu 1 --SiftExtraction.max_num_features "$MAX_FEATURES"; then + echo "⚠️ GPU SIFT failed (likely no GL context on this headless node) — retrying on CPU…" + rm -f "$DB" + colmap feature_extractor --database_path "$DB" --image_path "$IMG" \ + --SiftExtraction.use_gpu 0 --SiftExtraction.max_num_features "$MAX_FEATURES" +fi + +# 2) Matching — vocab-tree (sublinear). Exhaustive is O(N^2) and far too slow for 1000+-image scenes. +if [ ! -f "$VOCAB" ]; then + echo "⬇️ fetching vocab tree…" + wget -nc -q https://demuc.de/colmap/vocab_tree_flickr100K_words256K.bin -O "$VOCAB" +fi +echo "🔗 vocab_tree_matcher (GPU)…" +colmap vocab_tree_matcher --database_path "$DB" \ + --VocabTreeMatching.vocab_tree_path "$VOCAB" --SiftMatching.use_gpu 1 + +# 3) Report — want non-zero "verified pairs" (that's two_view_geometries, what GTSFM reads). +echo "📊 database_stats:" +colmap database_stats --database_path "$DB" +echo "✅ done → $DB" diff --git a/scripts/convert_wilsonkl_frontend.py b/scripts/convert_wilsonkl_frontend.py new file mode 100644 index 000000000..42646032a --- /dev/null +++ b/scripts/convert_wilsonkl_frontend.py @@ -0,0 +1,125 @@ +"""Convert a wilsonkl/SfM_Init (1DSfM) dataset release into a precomputed-frontend file. + +The 1DSfM benchmark ships the frontend every published method consumed: EGs.txt (verified +view-graph edges), coords.txt (per-image SIFT keypoints, original pixel frame), tracks.txt +(multi-view tracks as (image, key) pairs). This converts them into the npz consumed by +SceneOptimizer(precomputed_frontend_path=...), replacing retrieval + matching + two-view: + + keypoints are rescaled from the original frame to the loader frame (same + get_downsampling_factor_per_axis the loader applies, using the original dims recorded in + coords.txt headers), and per-edge verified correspondences are derived from the tracks + restricted to EGs edges. + +Usage: + python scripts/convert_wilsonkl_frontend.py --gt_dir --output \ + [--max_resolution 760] +""" +import argparse +import re +import time + +import numpy as np + +from gtsfm.utils.images import get_downsampling_factor_per_axis + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--gt_dir", required=True) + ap.add_argument("--output", required=True) + ap.add_argument("--max_resolution", type=int, default=760, + help="loader max_resolution (short side), must match the run config") + args = ap.parse_args() + t0 = time.time() + + # ---- coords.txt: per-image keypoints in ORIGINAL pixel frame + original dims ---- + kp = {} # img -> np.ndarray (N,2) in loader frame + dims = {} # img -> (loader_w, loader_h) expected at run time (EXIF-rotation guard) + header_re = re.compile(r"#index = (\d+),.*?keys = (\d+), px = ([\d.]+), py = ([\d.]+)") + cur_img, cur_rows, scale_uv = None, None, (1.0, 1.0) + + def flush(): + if cur_img is not None and cur_rows: + kp[cur_img] = np.array(cur_rows, dtype=np.float32) + + with open(f"{args.gt_dir}/coords.txt") as f: + for line in f: + if line.startswith("#index"): + flush() + m = header_re.search(line) + cur_img = int(m.group(1)) + px, py = float(m.group(3)), float(m.group(4)) + w0, h0 = 2 * px, 2 * py + su, sv, new_h, new_w = get_downsampling_factor_per_axis(int(h0), int(w0), args.max_resolution) + scale_uv = (su, sv) + dims[cur_img] = (new_w, new_h) + cur_rows = [] + else: + p = line.split() + if len(p) >= 3: + cur_rows.append((float(p[1]) * scale_uv[0], float(p[2]) * scale_uv[1])) + flush() + n_images = max(kp) + 1 + print(f"[+{time.time()-t0:.0f}s] coords: {len(kp)} images with keypoints (n_images={n_images})") + + # ---- EGs.txt: the verified edge set ---- + edges = set() + with open(f"{args.gt_dir}/EGs.txt") as f: + for line in f: + p = line.split() + if len(p) >= 2: + a, b = int(p[0]), int(p[1]) + edges.add((min(a, b), max(a, b))) + print(f"[+{time.time()-t0:.0f}s] EGs: {len(edges)} edges") + + # ---- tracks.txt -> per-edge correspondences (restricted to EGs edges) ---- + corr = {e: [] for e in edges} + n_tracks = 0 + with open(f"{args.gt_dir}/tracks.txt") as f: + first = f.readline() # track count header + for line in f: + p = line.split() + if not p: + continue + L = int(p[0]) + ms = [(int(p[1 + 2 * k]), int(p[2 + 2 * k])) for k in range(L)] + n_tracks += 1 + for x in range(L): + for y in range(x + 1, L): + (ia, ka), (ib, kb) = ms[x], ms[y] + if ia == ib: + continue + if ia > ib: + (ia, ka), (ib, kb) = (ib, kb), (ia, ka) + lst = corr.get((ia, ib)) + if lst is not None: + lst.append((ka, kb)) + corr = {e: np.array(c, dtype=np.int32) for e, c in corr.items() if len(c) >= 12} + n_corr = sum(len(c) for c in corr.values()) + print(f"[+{time.time()-t0:.0f}s] tracks: {n_tracks} -> {len(corr)} edges with >=12 " + f"correspondences ({n_corr} total)") + + # ---- flat-encode and save ---- + kp_offsets = np.zeros(n_images + 1, dtype=np.int64) + for i in range(n_images): + kp_offsets[i + 1] = kp_offsets[i] + (len(kp[i]) if i in kp else 0) + kp_flat = np.concatenate([kp[i] for i in range(n_images) if i in kp]) if kp else np.zeros((0, 2), np.float32) + E = sorted(corr) + edge_arr = np.array(E, dtype=np.int32) + corr_offsets = np.zeros(len(E) + 1, dtype=np.int64) + for k, e in enumerate(E): + corr_offsets[k + 1] = corr_offsets[k] + len(corr[e]) + corr_flat = np.concatenate([corr[e] for e in E]) if E else np.zeros((0, 2), np.int32) + img_dims = np.full((n_images, 2), -1, dtype=np.int32) + for i, (w, h) in dims.items(): + img_dims[i] = (w, h) + np.savez_compressed( + args.output, n_images=n_images, kp_offsets=kp_offsets, kp_flat=kp_flat, + edges=edge_arr, corr_offsets=corr_offsets, corr_flat=corr_flat, + max_resolution=args.max_resolution, img_dims=img_dims, + ) + print(f"[+{time.time()-t0:.0f}s] wrote {args.output}") + + +if __name__ == "__main__": + main() diff --git a/scripts/download_model_weights.sh b/scripts/download_model_weights.sh index da0dddb9f..42f63799f 100755 --- a/scripts/download_model_weights.sh +++ b/scripts/download_model_weights.sh @@ -1,5 +1,45 @@ # Download model weights for different modules available for use in GTSFM + +help() { + cat < +EOF + exit 0 +} + +case "$#" in + 0) + HF_TOKEN="" + ;; + 1) + if [[ "$1" == "--help" ]]; then + help + else + echo "Invalid argument" + exit 1 + fi + ;; + 2) + if [[ "$1" == "--hf_token" ]]; then + HF_TOKEN="$2" + else + echo "Invalid argument" + exit 1 + fi + ;; + *) + echo "Invalid number of arguments. Check --help" + exit 1 + ;; +esac + # Note: SuperPoint and SuperGlue code and checkpoints may *not* be used for commercial purposes #################### SuperPoint & SuperGlue ########################## @@ -45,11 +85,22 @@ wget $D2NET_CKPT_URL -O $D2NET_WEIGHTS_DIR/d2_tf.pth NETVLAD_WEIGHTS_DIR="./thirdparty/hloc/weights" NETVLAD_CKPT_URL="https://github.com/johnwlambert/gtsfm-datasets-mirror/releases/download/gerrard-hall-100/VGG16-NetVLAD-Pitts30K.mat" -mkdir $NETVLAD_WEIGHTS_DIR +mkdir -p $NETVLAD_WEIGHTS_DIR wget $NETVLAD_CKPT_URL -O $NETVLAD_WEIGHTS_DIR/VGG16-NetVLAD-Pitts30K.mat ##################### vggt ############################# VGGT_WEIGHTS_DIR="./thirdparty/vggt/weights" VGGT_CKPT_URL="https://huggingface.co/facebook/VGGT-1B/resolve/main/model.pt" -mkdir $VGGT_WEIGHTS_DIR +mkdir -p $VGGT_WEIGHTS_DIR wget $VGGT_CKPT_URL -O $VGGT_WEIGHTS_DIR/model.pt + +##################### vggt-omega ############################# +if [[ -z "$HF_TOKEN" ]]; then + echo "Hugging Face token not provided; skipping VGGT-Omega weights." +else + VGGT_OMEGA_WEIGHTS_DIR="./thirdparty/vggt-omega/weights" + mkdir -p $VGGT_OMEGA_WEIGHTS_DIR + hf download facebook/VGGT-Omega vggt_omega_1b_512.pt \ + --token "$HF_TOKEN" \ + --local-dir "$VGGT_OMEGA_WEIGHTS_DIR" +fi diff --git a/tests/test_angle_filter.py b/tests/test_angle_filter.py new file mode 100644 index 000000000..de1dd05d7 --- /dev/null +++ b/tests/test_angle_filter.py @@ -0,0 +1,52 @@ +"""Tests for the export-time triangulation-angle track filter.""" + +import numpy as np +from gtsam import Cal3Bundler, PinholeCameraCal3Bundler, Point3, Pose3, Rot3, SfmTrack + +from gtsfm.cluster_merging import filter_tracks_by_triangulation_angle +from gtsfm.common.gtsfm_data import GtsfmData + + +def _make_scene() -> GtsfmData: + """3 cameras on the x-axis; one low-parallax track and one wide-baseline track.""" + cal = Cal3Bundler(500.0, 0.0, 0.0, 0.0, 0.0) + scene = GtsfmData(number_images=3) + scene.add_camera(0, PinholeCameraCal3Bundler(Pose3(Rot3(), Point3(0.0, 0.0, 0.0)), cal)) + scene.add_camera(1, PinholeCameraCal3Bundler(Pose3(Rot3(), Point3(0.01, 0.0, 0.0)), cal)) + scene.add_camera(2, PinholeCameraCal3Bundler(Pose3(Rot3(), Point3(5.0, 0.0, 0.0)), cal)) + + # ~0.006 deg apex angle from cams {0,1}: depth-unconstrained. + low_parallax = SfmTrack(np.array([0.0, 0.0, 100.0])) + low_parallax.addMeasurement(0, np.array([0.0, 0.0])) + low_parallax.addMeasurement(1, np.array([-0.05, 0.0])) + + # ~28 deg apex angle from cams {0,2}: well-constrained. + wide = SfmTrack(np.array([2.5, 0.0, 10.0])) + wide.addMeasurement(0, np.array([125.0, 0.0])) + wide.addMeasurement(2, np.array([-125.0, 0.0])) + + assert scene.add_track(low_parallax) + assert scene.add_track(wide) + return scene + + +def test_low_parallax_track_dropped_wide_kept() -> None: + scene = _make_scene() + filtered = filter_tracks_by_triangulation_angle(scene, min_angle_deg=1.5) + assert filtered.number_tracks() == 1 + kept = filtered.get_track(0) + np.testing.assert_allclose(np.asarray(kept.point3()), [2.5, 0.0, 10.0]) + # Cameras are untouched: this is a track filter, not a camera filter. + assert filtered.get_valid_camera_indices() == scene.get_valid_camera_indices() + + +def test_zero_threshold_is_identity() -> None: + scene = _make_scene() + assert filter_tracks_by_triangulation_angle(scene, min_angle_deg=0.0) is scene + + +def test_all_tracks_survive_high_parallax() -> None: + scene = _make_scene() + # Both tracks clear a 0.001 deg bar. + filtered = filter_tracks_by_triangulation_angle(scene, min_angle_deg=0.001) + assert filtered.number_tracks() == 2 diff --git a/tests/test_cluster_ba_off.py b/tests/test_cluster_ba_off.py new file mode 100644 index 000000000..14b3e544b --- /dev/null +++ b/tests/test_cluster_ba_off.py @@ -0,0 +1,71 @@ +"""Tests for the run_bundle_adjustment=False path of _run_cluster_ba.""" + +import sys +import types + +import numpy as np +from gtsam import Cal3Bundler, PinholeCameraCal3Bundler, Point3, Pose3, Rot3, SfmTrack + +# cluster_vggt imports the VGGT model stack at module level; stub it so this CPU-only unit +# test (which only exercises the BA wrapper) can import without the vggt package installed. +if "vggt" not in sys.modules: + for name in ("vggt", "vggt.models", "vggt.models.vggt", "vggt.utils", + "vggt.utils.geometry", "vggt.utils.helper", "vggt.utils.load_fn", + "vggt.utils.pose_enc"): + sys.modules.setdefault(name, types.ModuleType(name)) + sys.modules["vggt.models.vggt"].VGGT = object + sys.modules["vggt.utils.geometry"].unproject_depth_map_to_point_map = lambda *a, **k: None + sys.modules["vggt.utils.helper"].randomly_limit_trues = lambda *a, **k: None + sys.modules["vggt.utils.load_fn"].load_and_preprocess_images_square = lambda *a, **k: None + sys.modules["vggt.utils.pose_enc"].pose_encoding_to_extri_intri = lambda *a, **k: None + +from gtsfm.bundle.bundle_adjustment import BundleAdjustmentOptions +from gtsfm.cluster_optimizer.cluster_vggt import _run_cluster_ba +from gtsfm.common.gtsfm_data import GtsfmData + + +def _make_scene() -> GtsfmData: + cal = Cal3Bundler(500.0, 0.0, 0.0, 0.0, 0.0) + scene = GtsfmData(number_images=2) + scene.add_camera(0, PinholeCameraCal3Bundler(Pose3(Rot3(), Point3(0.0, 0.0, 0.0)), cal)) + scene.add_camera(1, PinholeCameraCal3Bundler(Pose3(Rot3(), Point3(5.0, 0.0, 0.0)), cal)) + track = SfmTrack(np.array([2.5, 0.0, 10.0])) + track.addMeasurement(0, np.array([125.0, 0.0])) + track.addMeasurement(1, np.array([-125.0, 0.0])) + assert scene.add_track(track) + return scene + + +def test_ba_off_keeps_poses_and_tracks_verbatim() -> None: + scene = _make_scene() + post, pre = _run_cluster_ba( + scene, + ba_options=BundleAdjustmentOptions(), + pre_ba_max_reproj_error=14.0, + min_track_length=2, + run_bundle_adjustment=False, + ) + # Poses untouched (no optimizer ran), tracks retained (0px reproj passes the 14px pre-filter). + for i in (0, 1): + np.testing.assert_allclose( + np.asarray(post.get_camera(i).pose().translation()), + np.asarray(scene.get_camera(i).pose().translation()), + ) + assert post.number_tracks() == 1 + assert pre.number_tracks() == 1 + + +def test_ba_off_still_applies_pre_ba_filter() -> None: + scene = _make_scene() + bad = SfmTrack(np.array([2.5, 0.0, 10.0])) + bad.addMeasurement(0, np.array([300.0, 200.0])) # ~180px reproj error: junk + bad.addMeasurement(1, np.array([-300.0, -200.0])) + assert scene.add_track(bad) + post, _ = _run_cluster_ba( + scene, + ba_options=BundleAdjustmentOptions(), + pre_ba_max_reproj_error=14.0, + min_track_length=2, + run_bundle_adjustment=False, + ) + assert post.number_tracks() == 1 # junk track filtered, good one kept diff --git a/tests/test_create_v_corr_idxs_futures.py b/tests/test_create_v_corr_idxs_futures.py new file mode 100644 index 000000000..6eecf9eb3 --- /dev/null +++ b/tests/test_create_v_corr_idxs_futures.py @@ -0,0 +1,153 @@ +"""Unit tests for the parallel global two-view frontend (``create_v_corr_idxs_futures``). + +The parallel path chunks the pairs and runs ``create_v_corr_idxs_inline`` on each chunk across the Dask +worker pool, scattering the shared read-only inputs once. These tests assert it returns EXACTLY the same +``{(i1, i2): v_corr_idxs}`` dict as the serial inline reference for any chunking, and that the ``valid()`` +filter is honored. A deterministic stub estimator isolates the orchestration (chunking / scatter-as-blob / +merge) from real two-view geometry; cross-process worker pickling is exercised separately by end-to-end runs. + +Authors: Kathirvel Gounder +""" + +import unittest + +import numpy as np +from dask.distributed import Client, LocalCluster + +from gtsfm.common.keypoints import Keypoints +from gtsfm.two_view_estimator import create_v_corr_idxs_futures, create_v_corr_idxs_inline + + +class _StubResult: + """Minimal stand-in for TwoViewResult exposing only what the frontend reduction reads.""" + + def __init__(self, v_corr_idxs: np.ndarray, is_valid: bool) -> None: + self.v_corr_idxs = v_corr_idxs + self._is_valid = is_valid + + def valid(self) -> bool: + return self._is_valid + + +class _StubTwoViewEstimator: + """Deterministic stand-in for ``TwoViewEstimator.run_2view``. + + ``v_corr_idxs`` is derived from the pair's putative indices plus a per-pair offset, so inline and + parallel results can be compared exactly and a chunk can never be confused for another. Pairs whose + index sum is odd are marked invalid, exercising the ``valid()`` filter (invalid pairs must be dropped). + """ + + def run_2view(self, *, putative_corr_idxs, i1, i2, **kwargs) -> _StubResult: + is_valid = (i1 + i2) % 2 == 0 + v_corr_idxs = putative_corr_idxs + (i1 * 1000 + i2) + return _StubResult(v_corr_idxs, is_valid) + + +class _StubOneViewData: + """Per-image data with the attributes the frontend reads (all unused by the stub estimator).""" + + def __init__(self) -> None: + self.intrinsics = None + self.camera_gt = None + + +def _make_inputs(num_images: int = 8): + """Build deterministic keypoints, a fully-connected putative graph, priors, and per-view data.""" + keypoints_list = [Keypoints(np.full((5, 2), i, dtype=np.float32)) for i in range(num_images)] + putative_corr_idxs_dict = { + (i, j): np.arange(6, dtype=np.int32).reshape(3, 2) + (i + j) + for i in range(num_images) + for j in range(i + 1, num_images) + } + one_view_data_dict = {i: _StubOneViewData() for i in range(num_images)} + relative_pose_priors: dict = {} + return keypoints_list, putative_corr_idxs_dict, relative_pose_priors, one_view_data_dict + + +class TestCreateVCorrIdxsFutures(unittest.TestCase): + """Parallel two-view must equal the serial inline reference for any chunking.""" + + @classmethod + def setUpClass(cls) -> None: + # Threaded workers (processes=False): 2 workers, no cross-process pickling — deterministic and fast. + # This still exercises chunking, scatter-as-blob, and the as_completed merge, which is the new code. + cls.cluster = LocalCluster(n_workers=2, threads_per_worker=1, processes=False, dashboard_address=":0") + cls.client = Client(cls.cluster) + + @classmethod + def tearDownClass(cls) -> None: + cls.client.close() + cls.cluster.close() + + def _assert_same(self, expected: dict, actual: dict) -> None: + self.assertEqual(set(expected.keys()), set(actual.keys())) + for edge in expected: + np.testing.assert_array_equal(expected[edge], actual[edge]) + + def test_parallel_matches_inline_default_chunking(self) -> None: + """Default chunk sizing must reproduce the inline result exactly.""" + estimator = _StubTwoViewEstimator() + keypoints_list, putative, priors, one_view = _make_inputs() + + expected = create_v_corr_idxs_inline( + two_view_estimator=estimator, + keypoints_list=keypoints_list, + putative_corr_idxs_dict=putative, + relative_pose_priors=priors, + gt_scene_mesh=None, + one_view_data_dict=one_view, + ) + actual = create_v_corr_idxs_futures( + self.client, estimator, keypoints_list, putative, priors, None, one_view + ) + self._assert_same(expected, actual) + # Sanity: the valid() filter dropped the odd-sum pairs (so it is not a trivial pass-through). + self.assertTrue(all((i1 + i2) % 2 == 0 for (i1, i2) in actual)) + self.assertLess(len(actual), len(putative)) + + def test_parallel_matches_inline_tiny_chunks(self) -> None: + """Force many single-pair chunks (chunk_size=1) to stress the chunk/merge boundary.""" + estimator = _StubTwoViewEstimator() + keypoints_list, putative, priors, one_view = _make_inputs() + + expected = create_v_corr_idxs_inline( + two_view_estimator=estimator, + keypoints_list=keypoints_list, + putative_corr_idxs_dict=putative, + relative_pose_priors=priors, + gt_scene_mesh=None, + one_view_data_dict=one_view, + ) + actual = create_v_corr_idxs_futures( + self.client, estimator, keypoints_list, putative, priors, None, one_view, chunk_size=1 + ) + self._assert_same(expected, actual) + + def test_single_chunk_matches_inline(self) -> None: + """A chunk_size >= num_pairs (one chunk on one worker) must reproduce the inline result.""" + estimator = _StubTwoViewEstimator() + keypoints_list, putative, priors, one_view = _make_inputs() + + expected = create_v_corr_idxs_inline( + two_view_estimator=estimator, + keypoints_list=keypoints_list, + putative_corr_idxs_dict=putative, + relative_pose_priors=priors, + gt_scene_mesh=None, + one_view_data_dict=one_view, + ) + actual = create_v_corr_idxs_futures( + self.client, estimator, keypoints_list, putative, priors, None, one_view, chunk_size=10_000 + ) + self._assert_same(expected, actual) + + def test_empty_graph_returns_empty(self) -> None: + """No pairs → empty dict, no tasks submitted.""" + estimator = _StubTwoViewEstimator() + keypoints_list, _, priors, one_view = _make_inputs() + actual = create_v_corr_idxs_futures(self.client, estimator, keypoints_list, {}, priors, None, one_view) + self.assertEqual(actual, {}) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_gid_merge.py b/tests/test_gid_merge.py new file mode 100644 index 000000000..b1fc3be55 --- /dev/null +++ b/tests/test_gid_merge.py @@ -0,0 +1,263 @@ +"""Unit tests for global-track-ID merge anchoring (gid-index sidecar). + +Two clusters that triangulated the SAME global tracks from DISJOINT camera sets must be matchable by +global identity (the shared-camera matcher finds nothing there), and the MERGE_GUARD must drop a large +child whose ID-matched correspondences are too few to constrain its 7-DoF Sim3 seat. + +Authors: Kathirvel Gounder +""" + +import unittest + +import numpy as np +from gtsam import Cal3Bundler, PinholeCameraCal3Bundler, Point3, Pose3, Rot3, SfmTrack + +from gtsfm.cluster_merging import ( + _SCENE_GID_INDEX_ATTR, + _lookup_gid, + _select_gid_point_correspondences, + _track_gid, + _union_gid_indices, + annotate_scene_with_metadata, + build_measurement_gid_arrays, + merge_scenes_with_sim3_nonlinear, + slice_gid_index, +) +from gtsfm.common.gtsfm_data import GtsfmData +from gtsfm.common.sfm_track import SfmMeasurement, SfmTrack2d +from gtsfm.scene_optimizer import _dlt_triangulate, _ransac_dlt_resect + +NUM_IMAGES = 64 +PARENT_CAMS = [0, 1, 2] +CHILD_CAMS = [10, 11, 12] # DISJOINT from parent — no shared cameras anywhere. + + +def _uv(gid: int, cam: int) -> np.ndarray: + """Deterministic per-(track, camera) pixel; same values used for global tracks and scene tracks.""" + return np.array([13.0 * gid + cam + 0.25, 7.0 * gid + 2 * cam + 0.75]) + + +def _global_tracks(num: int) -> list[SfmTrack2d]: + """Global 2D tracks spanning BOTH camera sets (the cross-cluster edges made them).""" + tracks = [] + for gid in range(num): + meas = [SfmMeasurement(c, _uv(gid, c)) for c in PARENT_CAMS + CHILD_CAMS] + tracks.append(SfmTrack2d(measurements=meas)) + return tracks + + +def _scene(cams: list[int], gids: list[int], point_offset: float) -> GtsfmData: + """A reconstruction over ``cams`` containing one 3D track per gid (measurements = the global uvs).""" + data = GtsfmData(number_images=NUM_IMAGES) + for k, c in enumerate(cams): + pose = Pose3(Rot3(), np.array([k, 0.0, 0.0])) + data.add_camera(c, PinholeCameraCal3Bundler(pose, Cal3Bundler())) + for gid in gids: + track = SfmTrack(Point3(float(gid), point_offset, 0.0)) + for c in cams: + track.addMeasurement(c, _uv(gid, c)) + assert data.add_track(track) + return data + + +class TestGidIndex(unittest.TestCase): + def setUp(self) -> None: + self.tracks_2d = _global_tracks(60) + self.arrays = build_measurement_gid_arrays(self.tracks_2d) + + def test_pack_slice_lookup_roundtrip(self) -> None: + """Every measurement resolves to its gid through a camera-restricted slice.""" + parent_index = slice_gid_index(*self.arrays, set(PARENT_CAMS)) + child_index = slice_gid_index(*self.arrays, set(CHILD_CAMS)) + self.assertIsNotNone(parent_index) + for gid in (0, 7, 59): + self.assertEqual(_lookup_gid(parent_index, 0, _uv(gid, 0)), gid) + self.assertEqual(_lookup_gid(child_index, 11, _uv(gid, 11)), gid) + # A camera outside the slice must not resolve. + self.assertEqual(_lookup_gid(parent_index, 10, _uv(3, 10)), -1) + + def test_correspondences_across_disjoint_cameras(self) -> None: + """Same global tracks triangulated from disjoint camera sets match by identity.""" + parent = _scene(PARENT_CAMS, list(range(60)), point_offset=0.0) + child = _scene(CHILD_CAMS, list(range(60)), point_offset=100.0) + parent_index = slice_gid_index(*self.arrays, set(PARENT_CAMS)) + child_index = slice_gid_index(*self.arrays, set(CHILD_CAMS)) + + pairs = _select_gid_point_correspondences(parent, child, parent_index, child_index, max_correspondences=100) + self.assertEqual(len(pairs), 60) + # Pairs must join the SAME physical point: x-coords encode the gid on both sides. + for p_pt, c_pt in pairs: + self.assertAlmostEqual(p_pt[0], c_pt[0]) + self.assertAlmostEqual(p_pt[1] + 100.0, c_pt[1]) + + def test_track_gid_and_union(self) -> None: + parent_index = slice_gid_index(*self.arrays, set(PARENT_CAMS)) + child_index = slice_gid_index(*self.arrays, set(CHILD_CAMS)) + union = _union_gid_indices(parent_index, child_index) + parent = _scene(PARENT_CAMS, [5], point_offset=0.0) + child = _scene(CHILD_CAMS, [5], point_offset=100.0) + # Both scenes' copies of global track 5 resolve to the same gid through the union index. + self.assertEqual(_track_gid(parent.get_track(0), union), 5) + self.assertEqual(_track_gid(child.get_track(0), union), 5) + + def test_track_gid_majority_vote(self) -> None: + """A track whose FIRST measurement's pixel bin collides to a different gid still resolves to the + majority gid (the old first-hit identity returned the collided gid -> phantom correspondences).""" + tracks_2d = [ + # gid 0: a single measurement in camera 0 whose bin (10, 20) the test track will collide into. + SfmTrack2d(measurements=[SfmMeasurement(0, np.array([10.25, 20.75]))]), + # gid 1: measurements in cameras 1 and 2 — the test track's TRUE identity. + SfmTrack2d( + measurements=[ + SfmMeasurement(1, np.array([30.25, 40.75])), + SfmMeasurement(2, np.array([50.25, 60.75])), + ] + ), + ] + index = slice_gid_index(*build_measurement_gid_arrays(tracks_2d), {0, 1, 2}) + + track = SfmTrack(Point3(0.0, 0.0, 0.0)) + track.addMeasurement(0, np.array([10.9, 20.1])) # floors to bin (10, 20) -> collides to gid 0 + track.addMeasurement(1, np.array([30.5, 40.9])) # -> gid 1 + track.addMeasurement(2, np.array([50.7, 60.2])) # -> gid 1 + # Majority (2 votes gid 1 vs 1 collided vote gid 0) wins; first-hit would have returned 0. + self.assertEqual(_track_gid(track, index), 1) + + # And a track with no indexed measurements still resolves to -1. + unknown = SfmTrack(Point3(0.0, 0.0, 0.0)) + unknown.addMeasurement(0, np.array([9999.5, 9999.5])) + self.assertEqual(_track_gid(unknown, index), -1) + + def test_merge_guard_drops_large_weak_child(self) -> None: + """A >=30-camera child with too few ID correspondences is dropped (returns parent unchanged).""" + parent = _scene(PARENT_CAMS, list(range(60)), point_offset=0.0) + big_cams = list(range(20, 55)) # 35 cameras, disjoint from parent + # Child shares only 5 global tracks with the parent -> 5 ID correspondences << 50 floor. + child = GtsfmData(number_images=NUM_IMAGES) + for k, c in enumerate(big_cams): + child.add_camera(c, PinholeCameraCal3Bundler(Pose3(Rot3(), np.array([k, 0.0, 0.0])), Cal3Bundler())) + for gid in range(5): + track = SfmTrack(Point3(float(gid), 100.0, 0.0)) + for c in big_cams[:3]: + track.addMeasurement(c, _uv(gid, c)) + self.assertTrue(child.add_track(track)) + + # Global tracks for the child's cameras exist too (so its index resolves). + tracks_2d = _global_tracks(60) + for gid in range(5): + for c in big_cams[:3]: + tracks_2d[gid].measurements.append(SfmMeasurement(c, _uv(gid, c))) + arrays = build_measurement_gid_arrays(tracks_2d) + annotate_scene_with_metadata(parent, None, None, slice_gid_index(*arrays, set(PARENT_CAMS))) + annotate_scene_with_metadata(child, None, None, slice_gid_index(*arrays, set(big_cams))) + + merged = merge_scenes_with_sim3_nonlinear(parent, [child]) + # Guard dropped the only child -> parent returned as-is. + self.assertEqual(merged.number_tracks(), parent.number_tracks()) + self.assertEqual(set(merged.get_valid_camera_indices()), set(PARENT_CAMS)) + + def test_no_sidecar_runs_legacy_path(self) -> None: + """Without gid sidecars (enable_gid_merge_anchoring=False), the merge is the R3-baseline legacy + path: shared-camera matching, no corr-floor guard — a large low-corr child is KEPT, not dropped.""" + shared = list(range(30, 46)) + parent = _scene(PARENT_CAMS + shared, list(range(30)), point_offset=0.0) + child_cams = shared + list(range(46, 62)) # 32 cams, low legacy correspondences + child = _scene(child_cams, list(range(60)), point_offset=0.0) + # No annotate_scene_with_metadata -> no sidecars -> id_mode False everywhere. + merged = merge_scenes_with_sim3_nonlinear(parent, [child]) + self.assertTrue(set(child_cams).issubset(set(merged.get_valid_camera_indices()))) + + def test_structureless_child_dropped_any_size(self) -> None: + """A 0-track child is never merged, even with shared cameras (nothing can anchor or verify it).""" + parent = _scene(PARENT_CAMS, list(range(20)), point_offset=0.0) + child = GtsfmData(number_images=NUM_IMAGES) + for k, c in enumerate(PARENT_CAMS + [20, 21]): # shares ALL parent cameras, but zero tracks + child.add_camera(c, PinholeCameraCal3Bundler(Pose3(Rot3(), np.array([k, 0.0, 0.0])), Cal3Bundler())) + merged = merge_scenes_with_sim3_nonlinear(parent, [child]) + self.assertEqual(set(merged.get_valid_camera_indices()), set(PARENT_CAMS)) + + def test_overlap_escape_keeps_large_child_with_strong_camera_anchor(self) -> None: + """A large child with >=15 shared cameras is KEPT despite low ID-corr (measured: 21-shared seats fine). + + Reproduces the gid-run regression where a 50-cam child with 21 shared cameras and 25 correspondences + was dropped, costing Nc with no accuracy gain. + """ + shared = list(range(30, 46)) # 16 shared cameras + parent = _scene(PARENT_CAMS + shared, list(range(30)), point_offset=0.0) + child_cams = shared + list(range(46, 62)) # 32 cams total, 16 shared + child = _scene(child_cams, list(range(5)), point_offset=0.0) # only 5 MATCHING tracks (< 50 corr floor) + # Give the child real structure (>=50 tracks for the escape's structure floor) whose measurements + # do NOT resolve in the gid index (offset uvs) -> correspondences stay at 5. + for extra in range(60): + track = SfmTrack(Point3(float(extra), -50.0, 5.0)) + for c in child_cams[:4]: + track.addMeasurement(c, _uv(extra, c) + np.array([5000.0, 5000.0])) + self.assertTrue(child.add_track(track)) + + # Global tracks must SPAN this test's cameras so both slices resolve (id_mode active). + tracks_2d = [] + for gid in range(30): + meas = [SfmMeasurement(c, _uv(gid, c)) for c in PARENT_CAMS + shared + list(range(46, 62))] + tracks_2d.append(SfmTrack2d(measurements=meas)) + arrays = build_measurement_gid_arrays(tracks_2d) + annotate_scene_with_metadata(parent, None, None, slice_gid_index(*arrays, set(PARENT_CAMS + shared))) + annotate_scene_with_metadata(child, None, None, slice_gid_index(*arrays, set(child_cams))) + + merged = merge_scenes_with_sim3_nonlinear(parent, [child]) + # Escape clause: child kept (its cameras present in the merged scene). + self.assertTrue(set(child_cams).issubset(set(merged.get_valid_camera_indices()))) + + +class TestBoundaryRecoveryGeometry(unittest.TestCase): + """Unit tests for the boundary-recovery DLT helpers (ports of the offline-validated exp5 recipe).""" + + def test_dlt_triangulate_recovers_known_point(self) -> None: + """3 posed cameras observing a known 3D point: DLT recovers it to <1e-6 with ~0 reprojection.""" + f, px, py = 500.0, 320.0, 240.0 + x_true = np.array([0.5, 0.5, 10.0]) + centers = [np.array([0.0, 0.0, 0.0]), np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 0.0])] + posed, intrinsics, measurements = {}, {}, [] + for i, center in enumerate(centers): + r_w2c = np.eye(3) + t_w2c = -r_w2c @ center + posed[i] = (r_w2c, t_w2c) + intrinsics[i] = (f, 0.0, 0.0, px, py) + p = r_w2c @ x_true + t_w2c + measurements.append((i, p[0] / p[2] * f + px, p[1] / p[2] * f + py)) + + x_rec, err = _dlt_triangulate(measurements, posed, intrinsics) + self.assertIsNotNone(x_rec) + np.testing.assert_allclose(x_rec, x_true, atol=1e-6) + self.assertLess(err, 1e-6) + + def test_ransac_dlt_resect_recovers_known_camera(self) -> None: + """Noise-free observations of 60 known 3D points: resection recovers the exact w2c pose.""" + rng = np.random.default_rng(42) + struct = {g: rng.uniform([-2.0, -2.0, 4.0], [2.0, 2.0, 8.0]) for g in range(60)} + theta = 0.3 + r_true = np.array( + [ + [np.cos(theta), 0.0, np.sin(theta)], + [0.0, 1.0, 0.0], + [-np.sin(theta), 0.0, np.cos(theta)], + ] + ) + c_true = np.array([0.5, -0.3, -2.0]) + t_true = -r_true @ c_true + f, px, py = 600.0, 400.0, 300.0 + observations = [] + for g, point in struct.items(): + p = r_true @ point + t_true + observations.append((g, p[0] / p[2] * f + px, p[1] / p[2] * f + py)) + + result = _ransac_dlt_resect(observations, (f, 0.0, 0.0, px, py), struct) + self.assertIsNotNone(result) + r_rec, t_rec, c_rec, num_inliers = result + self.assertEqual(num_inliers, 60) + np.testing.assert_allclose(r_rec, r_true, atol=1e-6) + np.testing.assert_allclose(t_rec, t_true, atol=1e-6) + np.testing.assert_allclose(c_rec, c_true, atol=1e-6) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_torch_twoway_matcher.py b/tests/test_torch_twoway_matcher.py new file mode 100644 index 000000000..cc568bf36 --- /dev/null +++ b/tests/test_torch_twoway_matcher.py @@ -0,0 +1,48 @@ +"""A/B: TorchTwoWayMatcher must reproduce the OpenCV TwoWayMatcher on realistic descriptors.""" + +import numpy as np + +from gtsfm.frontend.matcher.torch_twoway_matcher import TorchTwoWayMatcher +from gtsfm.frontend.matcher.twoway_matcher import TwoWayMatcher + + +def _sift_like(n, seed): + rng = np.random.RandomState(seed) + d = rng.rand(n, 128).astype(np.float32) + return d / np.linalg.norm(d, axis=1, keepdims=True) * 512.0 # SIFT-scale magnitudes + + +def _match_both(d1, d2, ratio=0.8): + kwargs = dict(keypoints_i1=None, keypoints_i2=None, im_shape_i1=(100, 100), im_shape_i2=(100, 100)) + a = TwoWayMatcher(ratio_test_threshold=ratio).match(descriptors_i1=d1, descriptors_i2=d2, **kwargs) + b = TorchTwoWayMatcher(ratio_test_threshold=ratio).match(descriptors_i1=d1, descriptors_i2=d2, **kwargs) + return a, b + + +def test_matches_opencv_on_correlated_descriptors() -> None: + d1 = _sift_like(600, 0) + noise = _sift_like(600, 1) * 0.02 + d2 = np.vstack([d1[:300] + noise[:300], _sift_like(300, 2)]).astype(np.float32) + a, b = _match_both(d1, d2) + set_a = set(map(tuple, a.tolist())) + set_b = set(map(tuple, b.tolist())) + assert len(set_a) > 100 # sanity: real matches exist + jaccard = len(set_a & set_b) / max(len(set_a | set_b), 1) + assert jaccard > 0.99, f"jaccard {jaccard}: torch and opencv matchers disagree" + + +def test_no_ratio_test_mode() -> None: + d1 = _sift_like(200, 3) + d2 = np.vstack([d1[:150], _sift_like(50, 4)]).astype(np.float32) + a, b = _match_both(d1, d2, ratio=None) + set_a = set(map(tuple, a.tolist())) + set_b = set(map(tuple, b.tolist())) + jaccard = len(set_a & set_b) / max(len(set_a | set_b), 1) + assert jaccard > 0.99 + + +def test_degenerate_single_descriptor() -> None: + d1 = _sift_like(50, 5) + d2 = _sift_like(1, 6) + _, b = _match_both(d1, d2) + assert b.size == 0 or b.shape[1] == 2 diff --git a/tests/test_twoway_matcher_degenerate.py b/tests/test_twoway_matcher_degenerate.py new file mode 100644 index 000000000..0b2f8680e --- /dev/null +++ b/tests/test_twoway_matcher_degenerate.py @@ -0,0 +1,36 @@ +"""Regression: ratio-test matching must not crash when one side has <2 usable descriptors +(near-featureless internet images make knnMatch(k=2) return short candidate lists).""" + +import numpy as np + +from gtsfm.frontend.matcher.twoway_matcher import TwoWayMatcher + + +def _match(matcher, d1, d2): + return matcher.match( + keypoints_i1=None, + keypoints_i2=None, + descriptors_i1=d1, + descriptors_i2=d2, + im_shape_i1=(100, 100), + im_shape_i2=(100, 100), + ) + + +def test_single_train_descriptor_does_not_crash() -> None: + matcher = TwoWayMatcher(ratio_test_threshold=0.8) + rng = np.random.RandomState(0) + d1 = rng.rand(50, 128).astype(np.float32) + d2 = rng.rand(1, 128).astype(np.float32) # 1 descriptor -> knnMatch returns 1-tuples + result = _match(matcher, d1, d2) + assert result.size == 0 or result.shape[1] == 2 + + +def test_normal_matching_still_works() -> None: + matcher = TwoWayMatcher(ratio_test_threshold=0.8) + rng = np.random.RandomState(1) + d1 = rng.rand(60, 128).astype(np.float32) + # copy some rows so genuine matches exist + d2 = np.vstack([d1[:20] + 0.001, rng.rand(40, 128)]).astype(np.float32) + result = _match(matcher, d1, d2) + assert len(result) >= 10 diff --git a/tests/test_view_graph_calibration_robust.py b/tests/test_view_graph_calibration_robust.py new file mode 100644 index 000000000..31d3d97e9 --- /dev/null +++ b/tests/test_view_graph_calibration_robust.py @@ -0,0 +1,90 @@ +"""Tests for the robust (gated) Fetzer view-graph calibration.""" + +import numpy as np +from gtsam import Cal3Bundler + +from gtsfm.common.keypoints import Keypoints +from gtsfm.view_graph_estimator import view_graph_calibration as vgc + + +def _skew(t): + return np.array([[0, -t[2], t[1]], [t[2], 0, -t[0]], [-t[1], t[0], 0]]) + + +def _synthetic_pair(f=600.0, n=200, seed=0, planar=False): + """Correspondences from a known two-view geometry (f shared, pp at origin-offset 400,300).""" + rng = np.random.RandomState(seed) + K = np.array([[f, 0, 400.0], [0, f, 300.0], [0, 0, 1.0]]) + angle = 0.15 + R = np.array([ + [np.cos(angle), 0, np.sin(angle)], + [0, 1, 0], + [-np.sin(angle), 0, np.cos(angle)], + ]) + t = np.array([1.0, 0.15, 0.3]) + if planar: + X = np.column_stack([rng.uniform(-3, 3, n), rng.uniform(-2, 2, n), np.full(n, 8.0)]) + else: + X = np.column_stack([rng.uniform(-3, 3, n), rng.uniform(-2, 2, n), rng.uniform(5, 14, n)]) + x1 = (K @ X.T).T + x1 = x1[:, :2] / x1[:, 2:3] + X2 = (R @ X.T).T + t + x2 = (K @ X2.T).T + x2 = x2[:, :2] / x2[:, 2:3] + E = _skew(t) @ R + F = np.linalg.inv(K).T @ E @ np.linalg.inv(K) + return x1, x2, F / np.linalg.norm(F) + + +def test_focal_sanity_accepts_clean_F_rejects_random() -> None: + x1, x2, F = _synthetic_pair() + pp = np.array([400.0, 300.0]) + assert vgc.f_passes_focal_sanity(F, pp, pp, f_init=650.0) # near-truth init: interior min + rng = np.random.RandomState(3) + n_pass = sum( + vgc.f_passes_focal_sanity(rng.rand(3, 3), pp, pp, f_init=650.0) for _ in range(20) + ) + assert n_pass <= 2 # random matrices almost never imply a well-defined focal + + +def test_robust_F_planar_gate() -> None: + x1, x2, _ = _synthetic_pair(planar=True) + F, reason = vgc._estimate_fundamental_robust(x1, x2) + if vgc._HAS_POSELIB: + assert reason == "planar" and F is None + else: + assert F is not None or reason == "f_fail" + + +def test_calibrate_view_graph_recovers_focal() -> None: + """3 cams, true f=600, EXIF-ish init f=700: the gated pipeline should pull focals toward truth.""" + pairs = [(0, 1), (1, 2), (0, 2)] + keypoints = {} + v_corr = {} + store = {i: [] for i in range(3)} + for k, (a, b) in enumerate(pairs): + x1, x2, _ = _synthetic_pair(seed=k) + ia = np.arange(len(store[a]), len(store[a]) + len(x1)) + ib = np.arange(len(store[b]), len(store[b]) + len(x2)) + store[a].extend(x1.tolist()) + store[b].extend(x2.tolist()) + v_corr[(a, b)] = np.column_stack([ia, ib]).astype(np.int64) + for i in range(3): + keypoints[i] = Keypoints(coordinates=np.array(store[i])) + intr = {i: Cal3Bundler(700.0, 0.0, 0.0, 400.0, 300.0) for i in range(3)} + refined, _ = vgc.calibrate_view_graph(v_corr, keypoints, intr, min_correspondences=30) + focals = np.array([refined[i].fx() for i in range(3)]) + err0 = abs(700.0 - 600.0) / 600.0 + err1 = np.abs(focals - 600.0) / 600.0 + assert np.median(err1) < err0 * 0.35, f"focals {focals} did not converge toward 600" + + +def test_gates_off_matches_legacy_path() -> None: + x1, x2, _ = _synthetic_pair(seed=9) + keypoints = {0: Keypoints(coordinates=x1), 1: Keypoints(coordinates=x2)} + v_corr = {(0, 1): np.column_stack([np.arange(len(x1))] * 2).astype(np.int64)} + intr = {i: Cal3Bundler(650.0, 0.0, 0.0, 400.0, 300.0) for i in range(2)} + refined, _ = vgc.calibrate_view_graph( + v_corr, keypoints, intr, min_correspondences=30, use_robust_gates=False + ) + assert 0 in refined and 1 in refined # legacy path still runs end to end diff --git a/thirdparty/vggt-omega b/thirdparty/vggt-omega new file mode 160000 index 000000000..39a0cb8af --- /dev/null +++ b/thirdparty/vggt-omega @@ -0,0 +1 @@ +Subproject commit 39a0cb8af88554f15ddcb5354cd52bde588fa014 diff --git a/visualization/static/viewer.js b/visualization/static/viewer.js index e92b87235..7a1c641ad 100644 --- a/visualization/static/viewer.js +++ b/visualization/static/viewer.js @@ -16,23 +16,50 @@ class ColmapViewer { this.scene ); this.scene.activeCamera = this.camera; - this.camera.attachControl(canvas, true); this.camera.minZ = 0.01; // Near clipping plane this.camera.maxZ = 10000; // Far clipping plane - - // Z-up for interaction - this.camera.upVector = new BABYLON.Vector3(0, 0, -1); - - - // Inputs & limits tuned to avoid “locked” feeling - this.camera.angularSensibilityX = 1000; - this.camera.angularSensibilityY = 1000; - this.camera.panningSensibility = 1000; - this.camera.wheelPrecision = 40; - this.camera.lowerRadiusLimit = 0.1; + // Default Babylon Y-up. Instead of the old Z-up upVector + per-axis Y-flip hack (which left orbit + // fighting a mismatched up — the "hard gimbal lock"), a per-scene rotation maps the data's true up + // to +Y (see _computeUpAlignment) and is baked into every position at load. ArcRotate's up then + // matches the scene, so orbit feels natural. + + // Snappy feel (mirrors pipeline-viz). Lower sensibility = faster per-pixel; lower inertia = less drift. + this.camera.angularSensibilityX = 600; + this.camera.angularSensibilityY = 600; + this.camera.panningSensibility = 200; + this.camera.wheelPrecision = 8; + this.camera.inertia = 0.5; + this.camera.panningInertia = 0.5; + this.camera.lowerRadiusLimit = 0.001; this.camera.upperRadiusLimit = 10000; - this.camera.lowerBetaLimit = 0.001; - this.camera.upperBetaLimit = Math.TWO_PI - 0.001; + // Clamp beta OFF the poles (JS has no Math.TWO_PI; the old NaN limit let the camera flip through the + // poles). Keeping beta in (0, π) means orbit never crosses the up-axis singularity. + this.camera.lowerBetaLimit = 0.05; + this.camera.upperBetaLimit = Math.PI - 0.05; + // Exploration: zoom toward the cursor (not scene center) and support trackpad pinch. + this.camera.zoomToMouseLocation = true; + this.camera.useNaturalPinchZoom = true; + + // Clean, trackpad-friendly inputs: clear Babylon's defaults (their multi-touch twist misbehaves on + // macOS trackpads), then add only orbit/pan (pointers), wheel zoom, and arrow-key orbit. Order matters: + // clear() detaches, addX() registers, attachControl() binds DOM events. + this.camera.inputs.clear(); + this.camera.inputs.addPointers(); + this.camera.inputs.addMouseWheel(); + this.camera.inputs.addKeyboard(); + this.camera.keysUp = [38]; + this.camera.keysDown = [40]; + this.camera.keysLeft = [37]; + this.camera.keysRight = [39]; + this.camera.attachControl(canvas, true); + canvas.tabIndex = 0; + canvas.style.outline = "none"; + canvas.addEventListener("mouseenter", () => canvas.focus()); + + // Per-scene up-alignment (data → Y-up) + saved framing for the R key. + this.sceneRotation = null; + this.savedView = null; + this._cleanView = false; // H toggles this to hide all overlays for screenshots this.light = new BABYLON.HemisphericLight("H", new BABYLON.Vector3(0, 1, 0), this.scene); this.light.intensity = 0.85; @@ -71,6 +98,28 @@ class ColmapViewer { this.engine.runRenderLoop(() => this.scene.render()); window.addEventListener("resize", () => this.engine.resize()); + + // Camera hotkeys: R = reset framing, F = flip front/back, T = flip top/bottom, + // H = hide/show all UI (clean screenshot). Ignored while typing in a form field. + window.addEventListener("keydown", (e) => { + if (e.ctrlKey || e.metaKey || e.altKey) return; + const tag = (e.target && e.target.tagName) || ""; + if (tag === "INPUT" || tag === "TEXTAREA" || (e.target && e.target.isContentEditable)) return; + switch ((e.key || "").toLowerCase()) { + case "r": + if (this.savedView) { + const v = this.savedView; + this.camera.setTarget(new BABYLON.Vector3(v.center[0], v.center[1], v.center[2])); + this.camera.alpha = v.alpha; this.camera.beta = v.beta; this.camera.radius = v.radius; + } + break; + case "f": this.camera.alpha += Math.PI; break; + case "t": this.camera.beta = Math.PI - this.camera.beta; break; + case "h": this._toggleCleanView(); break; + default: return; + } + e.preventDefault(); + }); } setPointSize(px) { @@ -189,7 +238,14 @@ class ColmapViewer { _applyStatsVisibility() { if (!this.statsRoot) return; - this.statsRoot.style.display = this.statsVisible ? "" : "none"; + this.statsRoot.style.display = (this._cleanView || !this.statsVisible) ? "none" : ""; + } + + _toggleCleanView() { + // Hide ALL overlays (stats card + control bar) for an unobstructed money shot; press H again to restore. + this._cleanView = !this._cleanView; + this._applyStatsVisibility(); + this._applyHudMode(); } _applyStatsMode() { @@ -205,6 +261,7 @@ class ColmapViewer { if (!this.hudRoot) return; this.hudRoot.classList.remove("hud-hidden"); this.hudRoot.classList.toggle("hud-splat-mode", this.mode === "splat"); + this.hudRoot.style.display = this._cleanView ? "none" : ""; } _renderStats() { @@ -293,19 +350,35 @@ class ColmapViewer { this._setLoadingState({ message: "Parsing points…", progress: 0.35 }); } - const points = cachedPoints ?? this._parsePoints(pointsText ?? ""); + let points = cachedPoints ?? await this._parsePoints(pointsText ?? ""); if (needPoints && pointsUrl) { this._rememberParsed(this.parsedPointsCache, pointsUrl, points, this.parsedCacheLimit); } if (imagesUrl) { - this.cameras = cachedCameras ?? (imagesText ? this._parseCams(imagesText) : []); + this.cameras = cachedCameras ?? (imagesText ? await this._parseCams(imagesText) : []); if (needCameras) { this._rememberParsed(this.parsedCamsCache, imagesUrl, this.cameras, this.parsedCacheLimit); } } else { this.cameras = []; } + + // Compute the per-scene up-alignment (data → +Y) from cameras + landmarks, then bake it into every + // point position and camera center so ArcRotate's up matches the scene. Must run BEFORE building + // meshes; the parsers left everything in the raw data frame. + this._computeSceneOrientation(points, this.cameras); + // Build a NEW transformed array (do NOT mutate `points` — it may be the cached raw array, which + // would double-transform on a cache hit). Camera centers come from dataCenter, which is never + // mutated, so re-applying on a cache hit is idempotent. + points = points.map((p) => { + const r = this._applySceneRotation(p.x, p.y, p.z); + return { x: r[0], y: r[1], z: r[2], r: p.r, g: p.g, b: p.b }; + }); + for (const c of this.cameras) { + const r = this._applySceneRotation(c.dataCenter[0], c.dataCenter[1], c.dataCenter[2]); + c.center = new BABYLON.Vector3(r[0], r[1], r[2]); + } console.log('first camera center:', this.cameras[0]?.center); console.log('scene extent:', this._sceneExtent()); console.log(`Loaded: #points=${points.length}, #cams=${this.cameras.length}`); @@ -330,7 +403,7 @@ class ColmapViewer { this._updateCurrentImageName(); this._renderStats(); - this._autoFrame(); + this._autoFrame(points); this._setLoadingState({ progress: 1 }); } finally { this._setLoadingState({ active: false }); @@ -417,55 +490,183 @@ class ColmapViewer { // ------------------- parsing ------------------- - _parsePoints(text) { + async _parsePoints(text) { const pts = []; - for (const line of text.split("\n")) { - if (!line || line[0] === "#") continue; - const s = line.trim().split(/\s+/); - if (s.length >= 7) { - // flip Y for Babylon’s Y-up RH frame - pts.push({ - x: parseFloat(s[1]), y: -parseFloat(s[2]), z: parseFloat(s[3]), - r: parseInt(s[4]) / 255, g: parseInt(s[5]) / 255, b: parseInt(s[6]) / 255 - }); + const lines = text.split("\n"); + for (let i = 0; i < lines.length; i++) { + const line = lines[i]; + if (line && line[0] !== "#") { + const s = line.trim().split(/\s+/); + if (s.length >= 7) { + // Raw data frame; the per-scene up-alignment rotation is applied after parse. + pts.push({ + x: parseFloat(s[1]), y: parseFloat(s[2]), z: parseFloat(s[3]), + r: parseInt(s[4]) / 255, g: parseInt(s[5]) / 255, b: parseInt(s[6]) / 255 + }); + } } + // Yield to the browser every ~65k lines so parsing a large cloud never freezes the UI. + if ((i & 0xFFFF) === 0xFFFF) await new Promise(r => setTimeout(r)); } return pts; } - _parseCams(text) { + async _parseCams(text) { const cams = []; const lines = text.split("\n"); for (let i = 0; i < lines.length; i++) { const line = (lines[i] || "").trim(); - if (!line || line.startsWith("#")) continue; + if (!line || line.startsWith("#")) { + if ((i & 0xFFFF) === 0xFFFF) await new Promise(r => setTimeout(r)); + continue; + } const s = line.split(/\s+/); if (s.length >= 10) { const qw = parseFloat(s[1]), qx = parseFloat(s[2]), qy = parseFloat(s[3]), qz = parseFloat(s[4]); const tx = parseFloat(s[5]), ty = parseFloat(s[6]), tz = parseFloat(s[7]); - const qwc = new BABYLON.Quaternion(qx, qy, qz, qw); - const Rwc = new BABYLON.Matrix(); qwc.toRotationMatrix(Rwc); - const Rcw = Rwc.transpose(); - - const t = new BABYLON.Vector3(tx, ty, tz); - const centerWorld = BABYLON.Vector3.TransformCoordinates(t.scale(-1), Rcw); - - // Flip into viewer coords (Y flip) - const flip = BABYLON.Matrix.Scaling(1, -1, 1); - const center = new BABYLON.Vector3(centerWorld.x, -centerWorld.y, centerWorld.z); - const Rcam = flip.multiply(Rcw).multiply(flip); - + // Quaternion → 3x3 rotation R = cam_from_world (COLMAP convention), kept as a plain array. + const xx = qx*qx, yy = qy*qy, zz = qz*qz; + const xy = qx*qy, xz = qx*qz, yz = qy*qz; + const wx = qw*qx, wy = qw*qy, wz = qw*qz; + const R = [ + [1 - 2*(yy + zz), 2*(xy - wz), 2*(xz + wy)], + [2*(xy + wz), 1 - 2*(xx + zz), 2*(yz - wx)], + [2*(xz - wy), 2*(yz + wx), 1 - 2*(xx + yy)], + ]; + // Camera center in DATA coords = -R^T · t (scene up-alignment applied later, after we know it). + const dataCenter = [ + -(R[0][0]*tx + R[1][0]*ty + R[2][0]*tz), + -(R[0][1]*tx + R[1][1]*ty + R[2][1]*tz), + -(R[0][2]*tx + R[1][2]*ty + R[2][2]*tz), + ]; const name = s.length > 9 ? s.slice(9).join(" ") : ""; - cams.push({ center, R: Rcam, name }); + cams.push({ dataCenter, R, name, center: null }); // Skip 2D measurements line if present if (i + 1 < lines.length && /^\d/.test((lines[i + 1] || "").trim())) i++; } + if ((i & 0xFFFF) === 0xFFFF) await new Promise(r => setTimeout(r)); } return cams; } + // ------------------- scene up-alignment (ported from pipeline-viz) ------------------- + + _applySceneRotation(x, y, z) { + const R = this.sceneRotation; + if (!R) return [x, y, z]; + return [ + R[0][0] * x + R[0][1] * y + R[0][2] * z, + R[1][0] * x + R[1][1] * y + R[1][2] * z, + R[2][0] * x + R[2][1] * y + R[2][2] * z, + ]; + } + + _sampleXYZ(points, cap = 20000) { + // Sample points (data or transformed frame) into [x,y,z] arrays for robust stats. + const out = []; + if (!points || !points.length) return out; + const step = Math.max(1, Math.floor(points.length / cap)); + for (let i = 0; i < points.length; i += step) out.push([points[i].x, points[i].y, points[i].z]); + return out; + } + + _computeSceneOrientation(points, cams) { + // Compute (and store) the 3x3 rotation mapping the data's "up" to Babylon +Y. MUST run while points + // + cams are still in the raw data frame (before _applySceneRotation is baked in). + const camPositions = cams.map((c) => c.dataCenter); + const camRotations = cams.map((c) => c.R); + const landmarks = this._sampleXYZ(points); + this.sceneRotation = this._computeUpAlignment(camPositions, camRotations, landmarks); + } + + _computeUpAlignment(camPositions, camRotations, landmarks) { + let sceneUp = null; + // 1) Average camera image-up in world = -row1(cam_from_world); use if cameras agree. + if (camRotations && camRotations.length > 0) { + let sx = 0, sy = 0, sz = 0; + for (const R of camRotations) { sx += -R[1][0]; sy += -R[1][1]; sz += -R[1][2]; } + const n = camRotations.length; sx /= n; sy /= n; sz /= n; + const mag = Math.hypot(sx, sy, sz); + if (mag > 0.5) sceneUp = [sx / mag, sy / mag, sz / mag]; + } + // 2) Fallback: world axis of least variance over all positions. + if (!sceneUp) { + const all = landmarks && landmarks.length > 0 ? camPositions.concat(landmarks) : camPositions; + if (all.length < 3) return null; + let mx = 0, my = 0, mz = 0; + for (const [x, y, z] of all) { mx += x; my += y; mz += z; } + mx /= all.length; my /= all.length; mz /= all.length; + let cxx = 0, cyy = 0, czz = 0; + for (const [x, y, z] of all) { + const dx = x - mx, dy = y - my, dz = z - mz; + cxx += dx * dx; cyy += dy * dy; czz += dz * dz; + } + if (czz <= cxx && czz <= cyy) sceneUp = [0, 0, 1]; + else if (cyy <= cxx) sceneUp = [0, 1, 0]; + else sceneUp = [1, 0, 0]; + } + // 3) Flip so landmarks sit ABOVE cameras under the proposed up. + if (landmarks && landmarks.length > 0 && camPositions.length > 0) { + let lcx = 0, lcy = 0, lcz = 0; for (const [x, y, z] of landmarks) { lcx += x; lcy += y; lcz += z; } + lcx /= landmarks.length; lcy /= landmarks.length; lcz /= landmarks.length; + let ccx = 0, ccy = 0, ccz = 0; for (const [x, y, z] of camPositions) { ccx += x; ccy += y; ccz += z; } + ccx /= camPositions.length; ccy /= camPositions.length; ccz /= camPositions.length; + const dot = (lcx - ccx) * sceneUp[0] + (lcy - ccy) * sceneUp[1] + (lcz - ccz) * sceneUp[2]; + if (dot < 0) sceneUp = sceneUp.map((v) => -v); + } + // Build orthonormal basis [right, up, fwd] with up = sceneUp → maps sceneUp to +Y. + let ref = [1, 0, 0]; + if (Math.abs(ref[0] * sceneUp[0] + ref[1] * sceneUp[1] + ref[2] * sceneUp[2]) > 0.95) ref = [0, 1, 0]; + const xa = [ + sceneUp[1] * ref[2] - sceneUp[2] * ref[1], + sceneUp[2] * ref[0] - sceneUp[0] * ref[2], + sceneUp[0] * ref[1] - sceneUp[1] * ref[0], + ]; + const xn = Math.hypot(...xa); for (let i = 0; i < 3; i++) xa[i] /= (xn + 1e-12); + const za = [ + sceneUp[1] * xa[2] - sceneUp[2] * xa[1], + sceneUp[2] * xa[0] - sceneUp[0] * xa[2], + sceneUp[0] * xa[1] - sceneUp[1] * xa[0], + ]; + return [ + [xa[0], xa[1], xa[2]], + [sceneUp[0], sceneUp[1], sceneUp[2]], + [za[0], za[1], za[2]], + ]; + } + + _computeViewSetup(pts) { + // Robust framing: median center + 90th-percentile radius (x1.05). pts are [x,y,z] in Babylon frame. + if (!pts || pts.length === 0) return null; + const xs = pts.map((p) => p[0]).sort((a, b) => a - b); + const ys = pts.map((p) => p[1]).sort((a, b) => a - b); + const zs = pts.map((p) => p[2]).sort((a, b) => a - b); + const mid = (arr) => arr[Math.floor(arr.length / 2)]; + const center = [mid(xs), mid(ys), mid(zs)]; + const dists = pts + .map((p) => Math.hypot(p[0] - center[0], p[1] - center[1], p[2] - center[2])) + .sort((a, b) => a - b); + const range = (dists[Math.floor(dists.length * 0.9)] || 1.0) * 1.05; + return { center, range }; + } + + _computeAutoCameraAngles(pts, camPos) { + // Initial azimuth on the photographer side (camera cluster), 30° above horizontal. + if (!camPos || camPos.length === 0 || !pts || pts.length < 3) { + return { alpha: -Math.PI / 2, beta: Math.PI / 3 }; + } + let pmx = 0, pmz = 0; for (const p of pts) { pmx += p[0]; pmz += p[2]; } + pmx /= pts.length; pmz /= pts.length; + let cmx = 0, cmz = 0; for (const c of camPos) { cmx += c[0]; cmz += c[2]; } + cmx /= camPos.length; cmz /= camPos.length; + const vx = cmx - pmx, vz = cmz - pmz; + const norm = Math.hypot(vx, vz); + const alpha = norm > 1e-6 ? Math.atan2(vz, vx) : -Math.PI / 2; + return { alpha, beta: Math.PI / 3 }; + } + // ------------------- geometry ------------------- _centroidSample(points, n = 100) { @@ -482,6 +683,7 @@ class ColmapViewer { } async _createPCS(points) { + this._computeRobustBounds(points); // set robust center/radius before frustum sizing const pcs = new BABYLON.PointsCloudSystem( "pcs", { capacity: points.length, useColor: true }, @@ -1087,80 +1289,137 @@ class ColmapViewer { } } + _computeRobustBounds(points) { + // Robust scene center + radius from the parsed points: median center + 95th-percentile distance. + // Used to size frustums/camera dots and to reject out-of-range cameras, so a few stray points + // (common in messy higher-level merges) can't blow everything up. Computed from the points array + // (always available) rather than mesh vertex data (which a PointsCloudSystem may keep GPU-only). + const n = points.length; + if (!n) { this._robustCenter = new BABYLON.Vector3(0, 0, 0); this._robustRadius = 1.0; return; } + const step = Math.max(1, Math.floor(n / 20000)); // ~20k samples is plenty for robust stats + const xs = [], ys = [], zs = []; + for (let i = 0; i < n; i += step) { xs.push(points[i].x); ys.push(points[i].y); zs.push(points[i].z); } + const median = (a) => { a.sort((p, q) => p - q); return a[a.length >> 1]; }; + const cx = median(xs), cy = median(ys), cz = median(zs); + const dists = []; + for (let i = 0; i < n; i += step) { + const dx = points[i].x - cx, dy = points[i].y - cy, dz = points[i].z - cz; + dists.push(Math.sqrt(dx * dx + dy * dy + dz * dz)); + } + dists.sort((p, q) => p - q); + const p95 = dists[Math.min(dists.length - 1, Math.floor(dists.length * 0.95))] || 1.0; + this._robustCenter = new BABYLON.Vector3(cx, cy, cz); + this._robustRadius = p95 || 1.0; + } + _sceneExtent() { + // Robust extent = 2 x 95th-percentile radius (set by _computeRobustBounds at point-cloud build). + // Falls back to the raw bounding box only if robust bounds haven't been computed yet. + if (this._robustRadius) return 2 * this._robustRadius; if (!this.pointsMesh) return 1.0; const bb = this.pointsMesh.getBoundingInfo().boundingBox; - const size = bb.maximumWorld.subtract(bb.minimumWorld); - return Math.max(size.x, size.y, size.z) || 1.0; + const s = bb.maximumWorld.subtract(bb.minimumWorld); + return Math.max(s.x, s.y, s.z) || 1.0; } _createFrusta(cams) { - const size = this._sceneExtent(); - const frustumScale = 0.01 * size; // 1% of scene extent - const pivotDiam = 0.002 * size; // 0.2% sphere - + // One merged LineSystem for ALL camera frustums (previously 2 meshes per camera — a LineSystem + // + a Sphere — so ~1600 objects on an 800-camera scene). Camera-center dots become thin instances + // of a single sphere. Result: 2 meshes total regardless of camera count. Placement is identical to + // the old per-camera TransformNode (world = Compose(scale=1, rotation=R, translation=center)). + const size = this._sceneExtent(); // robust extent; also stashes _robustCenter/_robustRadius + const scale = 0.006 * size; // compact frustum (~0.6% of scene extent) for paper figures + const pivotDiam = 0.0015 * size; // small camera-center dot + + // Local frustum corners (camera looks down +Z), scaled — plain [x,y,z] arrays. + const cornersLocal = [ + [-0.4, -0.3, 1], [0.4, -0.3, 1], [0.4, 0.3, 1], [-0.4, 0.3, 1], + ].map(([x, y, z]) => [x * scale, y * scale, z * scale]); + + // Reject cameras with non-finite or wildly-out-of-range centers. Higher-level merges can place + // degenerate/unaligned cameras at garbage poses; drawing their frustums fills the screen with + // random lines. Anything past 12x the robust scene radius from the robust center is dropped. + const rc = this._robustCenter || new BABYLON.Vector3(0, 0, 0); + const rr = this._robustRadius || 0; + const lines = []; + const centers = []; + let skipped = 0; for (const c of cams) { - const node = new BABYLON.TransformNode("camNode", this.scene); - node.position.copyFrom(c.center); - node.rotationQuaternion = BABYLON.Quaternion.FromRotationMatrix(c.R); - node.parent = this.frustumGroup; - - const frustum = this._frustumLinesLocal(frustumScale); - frustum.parent = node; - frustum.renderingGroupId = 1; // draw after points - // If still hard to see, try: - frustum.alwaysSelectAsActiveMesh = true; - const m = new BABYLON.StandardMaterial("fmat", this.scene); - m.emissiveColor = new BABYLON.Color3(0, 0.5, 0); - frustum.material = m; - - const pivot = BABYLON.MeshBuilder.CreateSphere("camPivot", { diameter: pivotDiam }, this.scene); + const apex = c.center; + if (!isFinite(apex.x) || !isFinite(apex.y) || !isFinite(apex.z) || + (rr > 0 && BABYLON.Vector3.Distance(apex, rc) > 12 * rr)) { + skipped++; + continue; + } + centers.push(apex); + const R = c.R; // 3x3 cam_from_world (data frame) + const corners = cornersLocal.map(([lx, ly, lz]) => { + // world_from_cam · localOffset = R^T · local, then scene rotation → Babylon frame. + const wx = R[0][0] * lx + R[1][0] * ly + R[2][0] * lz; + const wy = R[0][1] * lx + R[1][1] * ly + R[2][1] * lz; + const wz = R[0][2] * lx + R[1][2] * ly + R[2][2] * lz; + const b = this._applySceneRotation(wx, wy, wz); + return new BABYLON.Vector3(apex.x + b[0], apex.y + b[1], apex.z + b[2]); + }); + lines.push([apex, corners[0]], [apex, corners[1]], [apex, corners[2]], [apex, corners[3]]); + lines.push([corners[0], corners[1]], [corners[1], corners[2]], [corners[2], corners[3]], [corners[3], corners[0]]); + } + if (skipped) console.warn(`[viz] hid ${skipped} camera(s) with out-of-range poses (likely bad merge)`); + + if (lines.length) { + const frusta = BABYLON.MeshBuilder.CreateLineSystem("frustumLines", { lines }, this.scene); + frusta.color = new BABYLON.Color3(0.1, 0.1, 0.1); // near-black wireframe (1DSfM / Snavely look) + frusta.isPickable = false; + frusta.renderingGroupId = 1; // draw after points + frusta.parent = this.frustumGroup; + } + + // Camera-center dots: one base sphere + a thin instance per camera (1 mesh for all). + if (centers.length) { + const base = BABYLON.MeshBuilder.CreateSphere("camPivotBase", { diameter: pivotDiam, segments: 6 }, this.scene); const pivotMat = new BABYLON.StandardMaterial("camPivotMat", this.scene); - pivotMat.emissiveColor = new BABYLON.Color3(0, 0, 0.5); + pivotMat.emissiveColor = new BABYLON.Color3(0.1, 0.1, 0.1); // match the dark frustums pivotMat.disableLighting = true; - pivot.material = pivotMat; - pivot.isPickable = false; - pivot.parent = node; + base.material = pivotMat; + base.isPickable = false; + base.renderingGroupId = 1; + base.parent = this.frustumGroup; + const matrices = new Float32Array(centers.length * 16); + const m = new BABYLON.Matrix(); + for (let i = 0; i < centers.length; i++) { + BABYLON.Matrix.TranslationToRef(centers[i].x, centers[i].y, centers[i].z, m); + m.copyToArray(matrices, i * 16); + } + base.thinInstanceSetBuffer("matrix", matrices, 16); } - this.setShowCams(this.showCams); - } - - _frustumLinesLocal(scale) { - const origin = BABYLON.Vector3.Zero(); - const corners = [ - new BABYLON.Vector3(-0.4, -0.3, 1), - new BABYLON.Vector3(0.4, -0.3, 1), - new BABYLON.Vector3(0.4, 0.3, 1), - new BABYLON.Vector3(-0.4, 0.3, 1), - ].map(v => v.scale(scale)); - - const lines = [ - [origin, corners[0]], [origin, corners[1]], [origin, corners[2]], [origin, corners[3]], - [corners[0], corners[1]], [corners[1], corners[2]], [corners[2], corners[3]], [corners[3], corners[0]], - ]; - const frustum = BABYLON.MeshBuilder.CreateLineSystem("frustumLines", { lines }, this.scene); - frustum.color = new BABYLON.Color3(1, 0.45, 0.25); - frustum.isPickable = false; - frustum.renderingGroupId = 1; // draw after points (optional) - return frustum; + this.setShowCams(this.showCams); } // ------------------- camera framing ------------------- - _autoFrame() { - // look-at: sampled centroid if we have it + _autoFrame(points) { + // Robust framing: median center + 90th-percentile range, camera placed on the photographer side at + // 30° elevation. Saves the view for the R (reset) key. Falls back to a bbox fit if no points. + const pts = this._sampleXYZ(points); + const setup = this._computeViewSetup(pts); + if (setup) { + const camPos = this.cameras.map((c) => [c.center.x, c.center.y, c.center.z]); + const angles = this._computeAutoCameraAngles(pts, camPos); + this.sceneCenter = new BABYLON.Vector3(setup.center[0], setup.center[1], setup.center[2]); + this.camera.setTarget(this.sceneCenter.clone()); + this.camera.radius = setup.range * 2.5; + this.camera.alpha = angles.alpha; + this.camera.beta = angles.beta; + this.savedView = { + center: setup.center.slice(), alpha: angles.alpha, beta: angles.beta, radius: setup.range * 2.5, + }; + this._updateCurrentImageName(); + return; + } const center = this.sceneCenter || new BABYLON.Vector3(0, 0, 0); this.camera.setTarget(center); - - if (this.cameras.length > 0) { - const pos = this.cameras[this.camIndex].center; - this.camera.setPosition(pos); - // recompute alpha/beta/radius from pos/target so orbit feels natural - this.camera.rebuildAnglesAndRadius?.(); - this._updateCurrentImageName(); - } else if (this.pointsMesh) { - // simple fit by radius if no cameras + if (this.pointsMesh) { const bb = this.pointsMesh.getBoundingInfo().boundingBox; const size = Math.max( bb.maximumWorld.x - bb.minimumWorld.x, @@ -1179,9 +1438,8 @@ class ColmapViewer { this.camera.setTarget(center); this.camera.setPosition(pos); this.camera.rebuildAnglesAndRadius?.(); - const R = this.cameras[this.camIndex].R; - const upWorld = BABYLON.Vector3.TransformCoordinates(new BABYLON.Vector3(0, 1, 0), R); - this.camera.upVector = upWorld; + // (Removed: slamming camera.upVector to the selected camera's up caused a disorienting + // orientation snap on every fly-to. Keeping the global up makes orbit stay consistent.) this._updateCurrentImageName(); this._renderStats(); }