diff --git a/visualization/hierarchy-web/README.md b/visualization/hierarchy-web/README.md new file mode 100644 index 000000000..9b1775e70 --- /dev/null +++ b/visualization/hierarchy-web/README.md @@ -0,0 +1,208 @@ +# Hierarchical Clustering Visualization + +Interactive Three.js web visualization of the GTSfM hierarchical clustering pipeline for 3D reconstruction. The visualization shows the complete end-to-end process of how individual VGGT cluster reconstructions are progressively merged into a complete building model. + +## Quick Start + +### 1. Place GTSfM results + +Copy the GTSfM `results/` output folder into this directory: + +``` +visualization/hierarchy-web/ +├── data/ +│ └── gerrard-hall-vggt/ +│ └── results/ +│ ├── vggt/ # Root VGGT cluster +│ │ ├── points3D.txt +│ │ ├── images.txt +│ │ └── cameras.txt +│ ├── C_1/vggt/ # Leaf cluster +│ ├── C_2/vggt/ # Leaf cluster +│ ├── ... +│ ├── C_4/merged/ # Intermediate merge +│ └── merged/ # Final merged reconstruction +│ ├── points3D.txt +│ ├── images.txt +│ └── cameras.txt +├── hierarchy-vggt.html +├── js/ +└── README.md +``` + +Each cluster folder must contain `points3D.txt` (COLMAP format with 3D coordinates and RGB color) and `images.txt` (camera poses with quaternions and translations). The tree structure is defined in `js/data-loader-vggt.js` — update `getStructure()` if the hierarchy changes. + +### 2. Start a local server + +```bash +cd visualization/hierarchy-web +python3 -m http.server 8080 +``` + +### 3. Open in browser + +Navigate to [http://localhost:8080/hierarchy-vggt.html](http://localhost:8080/hierarchy-vggt.html) + +## Controls + +| Button | Action | +|--------|--------| +| **Play/Pause** | Auto-advance through all timeline events | +| **Prev/Next** | Step through events one at a time | +| **Reset** | Return to the first event | +| **Record** | Start/stop recording the visualization as a `.webm` video | + +Mouse: drag to orbit, scroll to zoom, right-click drag to pan. Click anywhere on the timeline progress bar to jump to a specific event. + +## Recording + +Click the **Record** button to start capturing the visualization. The button turns red and pulses while recording. Click **Stop** to end the recording — the video file automatically downloads as a `.webm` file (timestamped filename) to your browser's default download location. + +--- + +## How the Visualization Works + +The visualization displays the hierarchical clustering process used by the GTSfM pipeline. The pipeline reconstructs a 3D scene by first running VGGT (Visual Geometry Grounded Transformer) on small clusters of images, then progressively merging neighboring clusters until a single complete reconstruction is produced. This visualization animates that process step by step. + +### What You See + +1. **Leaf clusters appear** — Individual VGGT reconstructions (point clouds) are placed on screen in allocated tiles +2. **Clusters merge** — When sibling clusters merge, individual 3D points smoothly fly from their child-cluster positions to their merged positions +3. **Final building** — The last merge event produces the complete 3D reconstruction of the building + +The visualization is driven by a **timeline of events**. Leaf placement events happen first (deepest clusters first, then shallower ones), followed by merge events (deepest merges first, up to the root). + +--- + +## Algorithms + +### 1. Scene Orientation (Camera-Based) + +**Problem**: The SfM reconstruction places the building in an arbitrary coordinate system determined by the pipeline's internal optimization. There is no guarantee that "up" in the reconstruction corresponds to "up" on screen. Without correction, the building appears sideways or at an arbitrary angle. + +**Solution**: We compute the correct orientation mathematically from the actual camera poses stored in `images.txt` (COLMAP format). + +**How it works**: + +1. **Parse camera extrinsics**: Each camera in `images.txt` has a quaternion `(qw, qx, qy, qz)` representing the rotation from world to camera frame, and a translation `(tx, ty, tz)` in the camera frame. + +2. **Compute camera world positions**: For each camera, the world-space position is `C = -R^T * t`, where `R` is the 3x3 rotation matrix built from the quaternion and `R^T` is its transpose (camera-to-world transform). + +3. **Find the "up" direction**: Each camera has an "up" direction in world space computed as `R^T * [0, -1, 0]` (negated because COLMAP's Y-axis points down). Averaging all camera up-vectors across the dataset gives the true vertical axis in the reconstruction's coordinate system. In practice, photographers hold cameras roughly level, so these vectors are highly consistent (measured consistency of 0.9757 out of 1.0 for the Gerrard Hall dataset). + +4. **Find the "front" direction**: The vector from the camera centroid (average position of all cameras) to the building centroid (average position of all 3D points) gives the dominant viewing direction. This is orthogonalized against the up-vector to ensure perpendicularity. + +5. **Build rotation matrix**: From the up and front vectors, an orthonormal basis is constructed using cross products: + - `scene_Y` = normalized up direction + - `scene_Z` = front direction, orthogonalized against scene_Y + - `scene_X` = cross(scene_Y, scene_Z) + + These three vectors form the rows of a 3x3 rotation matrix that transforms the reconstruction's world frame into Three.js screen space (Y-up, Z-toward-viewer). + +**Why this approach**: This is fully data-driven — no hardcoded rotation angles. When a new dataset is provided (different building, different camera arrangement), the orientation is automatically computed from that dataset's camera poses. + +**Implementation**: `loadCameraExtrinsics()` and `computeSceneOrientation()` in `js/data-loader-vggt.js`. + +--- + +### 2. Cluster Placement Algorithm (Squareness-Optimized Layout) + +**Problem**: The hierarchical tree of clusters must be arranged on screen so that (a) each cluster fills its allocated space well, (b) sibling clusters are adjacent (for natural merge animations), and (c) the screen is filled without excessive white space. + +**Solution**: A recursive rectangle subdivision algorithm that exhaustively searches all possible row/column groupings at each tree level and picks the one that maximizes the worst-case tile squareness. + +**How it works**: + +1. **Compute the root rectangle**: A 16:9 rectangle is sized based on the number of leaf clusters and the largest cluster's bounding sphere radius: + ``` + tileSize = maxRadius * 2.2 + totalArea = tileSize^2 * leafCount * 1.1 + ROOT_H = sqrt(totalArea / (16/9)) + ROOT_W = ROOT_H * 16/9 + ``` + +2. **Weight each child by leaf count**: At each internal node, children are assigned weights equal to their number of descendant leaf clusters. For example, at the root with children `[vggt(1), C_1(1), C_2(1), C_3(1), C_4/merged(3)]`, the weights are `[1, 1, 1, 1, 3]`. + +3. **Generate all possible partitions**: For `n` children, there are `2^(n-1)` ways to partition them into contiguous groups (each of the `n-1` gaps between children is either a row/column break or not). For 5 children, that's 16 possible groupings. + +4. **Evaluate each partition in both orientations**: Each partition is tested as both "rows" (groups stacked vertically) and "columns" (groups side by side horizontally) — giving `2^(n-1) * 2` total candidates. + + For each candidate: + - Each group (row/column) receives height/width proportional to its total weight + - Within each group, children receive width/height proportional to their individual weights + - The **squareness** of each child's tile is computed as `min(width, height) / max(width, height)` (1.0 = perfect square, 0.0 = infinitely elongated) + - The **worst-case squareness** across all tiles is recorded + +5. **Pick the best**: The partition with the highest worst-case squareness wins. This ensures no tile is excessively elongated. + +6. **Recurse**: Each child's allocated rectangle is then recursively subdivided using the same algorithm for its own children. + +7. **Fit clusters into tiles**: Each leaf cluster's 3D point cloud is scaled to fit inside its tile: + ``` + fitDim = min(tile_width, tile_height) * 0.85 + fitScale = fitDim / (2 * cluster_radius) + ``` + The 0.85 factor provides a 15% margin so clusters don't touch tile edges. The division by `2 * radius` (diameter) ensures the full bounding sphere fits within the tile. + +**Example**: For the Gerrard Hall dataset with weights `[1,1,1,1,3]`, the winning layout is two rows — `[vggt, C_1, C_2]` on top and `[C_3, C_4/merged]` on bottom — with a worst-case tile squareness of ~0.43, compared to ~0.25 for the naive all-columns approach. + +**Implementation**: `assignLeafTiles()` in `js/layout-engine-squareness.js`. + +--- + +### 3. Merge Animation (Per-Point Alpha Blending) + +**Problem**: When two or more child clusters merge into a parent cluster, the visualization needs to show this transition smoothly. The child point clouds and merged point cloud have different numbers of points and different point positions — some points exist only in children, some only in the merged result, and some correspond between both. + +**Solution**: Per-point interpolation where each point individually transitions from its child position to its merged position, with three categories of points handled differently. + +**How it works**: + +1. **Point matching**: Before any animation, a nearest-neighbor matching step establishes correspondence between child points and merged points. + - For each child point, the nearest merged point is found (brute-force nearest neighbor in 3D) + - Candidates are sorted by distance. A threshold is set at `4x the 90th percentile distance` — pairs below this threshold are considered matches + - A greedy 1-to-1 assignment ensures each merged point is matched to at most one child point + +2. **Three categories of points during a merge**: + - **Matched points** (~60-80% of points): These exist in both child and merged results. During the animation, each point smoothly interpolates from its child-cluster position to its merged position using cubic easing (`easeInOutCubic`). Three temporary `THREE.Points` objects are created — one for matched, one for child-only, one for merged-only. + - **Child-only points** (exist only in children, not in merged): These fly toward their nearest merged point and fade out slightly (`opacity 1 → 0.7`) during the transition. + - **Merged-only points** (exist only in merged result, not in children): These fly from their nearest child point to their final merged position, fading in during the transition. + +3. **Animation timeline**: Each merge transition lasts 2.5 seconds. The easing function `easeInOutCubic` produces smooth acceleration and deceleration. At the end of the transition, the temporary point clouds are disposed and the merged cluster's original point cloud is made visible. + +4. **Auto-play guard**: The auto-play system waits for the current merge animation to fully complete before advancing to the next event. This prevents race conditions where overlapping animations could produce incorrect visual states. + +**Implementation**: `playMergeTransition()` and `update()` in `js/animation-engine-squareness.js`, point matching in `computePointMatching()` in `js/data-loader-vggt.js`. + +--- + +### 4. The Final Visualization + +The complete visualization sequence for the Gerrard Hall dataset has **9 events**: + +- **Events 1-7** (leaf placements): The 7 leaf VGGT clusters appear one at a time in their allocated tiles, with a fade-in animation. Deeper clusters in the tree appear first. +- **Event 8** (intermediate merge): The 3 clusters in the C_4 subtree (`C_4/vggt`, `C_4_1/vggt`, `C_4_2/vggt`) merge into `C_4/merged`. Individual points fly from their child positions to their merged positions. +- **Event 9** (final merge): All remaining clusters (`vggt`, `C_1/vggt`, `C_2/vggt`, `C_3/vggt`, `C_4/merged`) merge into the root `merged` cluster, producing the complete 3D reconstruction of Gerrard Hall. The final point cloud is displayed centered on screen. + +The end result shows the fully reconstructed building with RGB colors from the original photographs, oriented upright and facing the viewer. + +--- + +## File Architecture + +| File | Purpose | +|------|---------| +| `hierarchy-vggt.html` | Main page with Three.js imports, UI controls, and CSS styling | +| `js/data-loader-vggt.js` | Loads `points3D.txt` and `images.txt`, defines tree structure, computes scene orientation and point matching | +| `js/layout-engine-squareness.js` | Recursive rectangle layout with exhaustive partition search for optimal tile squareness | +| `js/animation-engine-squareness.js` | Timeline event system with per-point merge interpolation and fade animations | +| `js/main-hierarchy-vggt.js` | App entry point: Three.js scene, camera, renderer, orbit controls, UI wiring, recording | +| `js/camera-engine.js` | Camera flythrough support for automated camera movements | +| `js/interaction-engine.js` | Mouse hover/click interaction with point cloud clusters | + +## Adapting to New Datasets + +1. Place the new GTSfM results in `data//results/` +2. Update `getStructure()` in `js/data-loader-vggt.js` to reflect the new cluster tree hierarchy +3. Update the data path in `loadPointCloud()` and `loadCameraExtrinsics()` +4. The orientation and layout algorithms will adapt automatically to the new data — no hardcoded values need changing diff --git a/visualization/hierarchy-web/hierarchy-vggt.html b/visualization/hierarchy-web/hierarchy-vggt.html new file mode 100644 index 000000000..ba94f2d25 --- /dev/null +++ b/visualization/hierarchy-web/hierarchy-vggt.html @@ -0,0 +1,324 @@ + + + + + + Gerrard Hall - VGGT Squareness Layout + + + +
+
+ +
+ +
Loading VGGT Point Clouds...
+
+ +
+
+
+

VGGT Squareness Layout ▼

+
+
Recursive rectangle subdivision with squareness-optimized cluster placement
+
VGGT PIPELINE
+
+
+ ← Back to Menu +
+ +
+
+ Timeline + Initializing... + ▼ +
+ +
+
+
+
+ +
+ + + + + +
+ +
+ Clusters: 0 | Points: 0 +
+
+
+
+
+ + + + + + + + diff --git a/visualization/hierarchy-web/js/animation-engine-squareness.js b/visualization/hierarchy-web/js/animation-engine-squareness.js new file mode 100644 index 000000000..0327370a1 --- /dev/null +++ b/visualization/hierarchy-web/js/animation-engine-squareness.js @@ -0,0 +1,426 @@ +import * as THREE from 'three'; + +/** + * Animation Engine with Per-Point Interpolation + * + * Merge transitions move individual points from their child-cluster + * positions to their merged-result positions, matching the approach + * from the Gemini point-set interpolation demo: + * - Matched points lerp smoothly (cubic easing) + * - Child-only points fade out in place + * - Merged-only points fade in at target + */ +export class SquarenessAnimationEngine { + constructor(clusters, layoutEngine, worldGroup) { + this.clusters = clusters; + this.layoutEngine = layoutEngine; + this.worldGroup = worldGroup; + this.mergeEvents = []; + this.activeAnimations = []; + this.transitionClouds = []; + this.mergeDuration = 2.5; + this.leafFadeDuration = 0.5; + } + + initTimeline() { + const treeNodes = this.layoutEngine.treeNodes; + if (!treeNodes || treeNodes.length === 0) { + console.warn("No tree nodes for timeline"); + return []; + } + + const leaves = []; + const merges = []; + for (const node of treeNodes) { + if (node.children.length === 0) leaves.push(node); + else merges.push(node); + } + + leaves.sort((a, b) => { + if (b.depth !== a.depth) return b.depth - a.depth; + return a.cluster.path.localeCompare(b.cluster.path); + }); + merges.sort((a, b) => { + if (b.depth !== a.depth) return b.depth - a.depth; + return a.cluster.path.localeCompare(b.cluster.path); + }); + + for (const node of leaves) { + this.mergeEvents.push({ + path: node.cluster.path, + cluster: node.cluster, + isLeaf: true, + children: [], + depth: node.depth + }); + } + for (const node of merges) { + this.mergeEvents.push({ + path: node.cluster.path, + cluster: node.cluster, + isLeaf: false, + children: node.children.map(c => c.cluster.path), + depth: node.depth + }); + } + + console.log(`Timeline: ${this.mergeEvents.length} events (${leaves.length} leaves + ${merges.length} merges)`); + return this.mergeEvents; + } + + cleanupTransitions() { + for (const cloud of this.transitionClouds) { + this.worldGroup.remove(cloud); + cloud.geometry.dispose(); + cloud.material.dispose(); + } + this.transitionClouds = []; + this.activeAnimations = this.activeAnimations.filter(a => a.type !== 'mergeTransition'); + } + + applyEventInstant(eventIndex) { + this.cleanupTransitions(); + this.activeAnimations = []; + + for (const cluster of this.clusters.values()) { + if (cluster.pointCloud) { + cluster.pointCloud.visible = false; + cluster.pointCloud.material.opacity = 1; + } + if (cluster.hierarchyPosition) { + cluster.group.position.copy(cluster.hierarchyPosition); + } + if (cluster.fitScale) { + cluster.group.scale.setScalar(cluster.fitScale); + } + } + + for (let i = 0; i <= eventIndex; i++) { + const evt = this.mergeEvents[i]; + if (!evt) continue; + const c = evt.cluster; + + if (evt.isLeaf) { + if (c.pointCloud) c.pointCloud.visible = true; + } else { + if (c.pointCloud) c.pointCloud.visible = true; + for (const childPath of evt.children) { + const child = this.clusters.get(childPath); + if (child && child.pointCloud) child.pointCloud.visible = false; + } + } + } + } + + playEvent(eventIndex, direction = 1) { + const evt = this.mergeEvents[eventIndex]; + if (!evt) return; + + if (direction > 0) { + if (evt.isLeaf) { + if (evt.cluster.pointCloud) { + evt.cluster.pointCloud.visible = true; + this.animateFadeIn(evt.cluster); + } + } else { + this.playMergeTransition(evt); + } + } else { + this.cleanupTransitions(); + if (evt.isLeaf) { + if (evt.cluster.pointCloud) this.animateFadeOut(evt.cluster); + } else { + if (evt.cluster.pointCloud) { + evt.cluster.pointCloud.visible = false; + evt.cluster.pointCloud.material.opacity = 1; + } + for (const childPath of evt.children) { + const child = this.clusters.get(childPath); + if (child && child.pointCloud) { + child.pointCloud.visible = true; + child.pointCloud.material.opacity = 1; + child.group.position.copy(child.hierarchyPosition); + if (child.fitScale) child.group.scale.setScalar(child.fitScale); + } + } + } + } + } + + playMergeTransition(evt) { + const merged = evt.cluster; + const matchData = merged.matchData; + + if (!matchData || !merged.pointCloud) { + if (merged.pointCloud) { + merged.pointCloud.visible = true; + this.animateFadeIn(merged); + } + for (const childPath of evt.children) { + const child = this.clusters.get(childPath); + if (child && child.pointCloud && child.pointCloud.visible) { + this.animateFallbackMerge(child, merged.hierarchyPosition); + } + } + return; + } + + const { matchedPairs, childOnlyPoints, mergedOnlyIndices } = matchData; + + for (const childPath of evt.children) { + const child = this.clusters.get(childPath); + if (child && child.pointCloud) child.pointCloud.visible = false; + } + merged.pointCloud.visible = false; + + // --- Matched points cloud --- + const mLen = matchedPairs.length; + const matchedStart = new Float32Array(mLen * 3); + const matchedEnd = new Float32Array(mLen * 3); + const matchedColors = new Float32Array(mLen * 3); + + for (let i = 0; i < mLen; i++) { + const pair = matchedPairs[i]; + const child = this.clusters.get(pair.childPath); + if (!child || !child.pointCloud) continue; + + this.writeWorldPos(child, pair.childIdx, matchedStart, i * 3); + this.writeWorldPos(merged, pair.mergedIdx, matchedEnd, i * 3); + this.writeColor(child, pair.childIdx, matchedColors, i * 3); + } + + const matchedGeom = new THREE.BufferGeometry(); + matchedGeom.setAttribute('position', new THREE.Float32BufferAttribute(matchedStart.slice(), 3)); + matchedGeom.setAttribute('color', new THREE.Float32BufferAttribute(matchedColors, 3)); + const matchedMat = this.makeTransitionMaterial(1.0); + const matchedCloud = new THREE.Points(matchedGeom, matchedMat); + + // --- Child-only cloud (fly to nearest merged point) --- + const coLen = childOnlyPoints.length; + const coStart = new Float32Array(coLen * 3); + const coEnd = new Float32Array(coLen * 3); + const coCol = new Float32Array(coLen * 3); + + for (let i = 0; i < coLen; i++) { + const cp = childOnlyPoints[i]; + const child = this.clusters.get(cp.childPath); + if (!child || !child.pointCloud) continue; + this.writeWorldPos(child, cp.childIdx, coStart, i * 3); + this.writeColor(child, cp.childIdx, coCol, i * 3); + if (cp.flyToMergedIdx !== undefined) { + this.writeWorldPos(merged, cp.flyToMergedIdx, coEnd, i * 3); + } else { + coEnd[i * 3] = coStart[i * 3]; + coEnd[i * 3 + 1] = coStart[i * 3 + 1]; + coEnd[i * 3 + 2] = coStart[i * 3 + 2]; + } + } + + const coGeom = new THREE.BufferGeometry(); + coGeom.setAttribute('position', new THREE.Float32BufferAttribute(coStart.slice(), 3)); + coGeom.setAttribute('color', new THREE.Float32BufferAttribute(coCol, 3)); + const coMat = this.makeTransitionMaterial(1.0); + const childOnlyCloud = new THREE.Points(coGeom, coMat); + + // --- Merged-only cloud (fly from nearest child point) --- + const moLen = mergedOnlyIndices.length; + const moStart = new Float32Array(moLen * 3); + const moEnd = new Float32Array(moLen * 3); + const moCol = new Float32Array(moLen * 3); + + for (let i = 0; i < moLen; i++) { + const entry = mergedOnlyIndices[i]; + const mIdx = typeof entry === 'object' ? entry.mergedIdx : entry; + this.writeWorldPos(merged, mIdx, moEnd, i * 3); + this.writeColor(merged, mIdx, moCol, i * 3); + if (typeof entry === 'object' && entry.flyFromChildPath) { + const srcChild = this.clusters.get(entry.flyFromChildPath); + if (srcChild && srcChild.pointCloud) { + this.writeWorldPos(srcChild, entry.flyFromChildIdx, moStart, i * 3); + } else { + moStart[i * 3] = moEnd[i * 3]; + moStart[i * 3 + 1] = moEnd[i * 3 + 1]; + moStart[i * 3 + 2] = moEnd[i * 3 + 2]; + } + } else { + moStart[i * 3] = moEnd[i * 3]; + moStart[i * 3 + 1] = moEnd[i * 3 + 1]; + moStart[i * 3 + 2] = moEnd[i * 3 + 2]; + } + } + + const moGeom = new THREE.BufferGeometry(); + moGeom.setAttribute('position', new THREE.Float32BufferAttribute(moStart.slice(), 3)); + moGeom.setAttribute('color', new THREE.Float32BufferAttribute(moCol, 3)); + const moMat = this.makeTransitionMaterial(0.0); + const mergedOnlyCloud = new THREE.Points(moGeom, moMat); + + this.worldGroup.add(matchedCloud); + this.worldGroup.add(childOnlyCloud); + this.worldGroup.add(mergedOnlyCloud); + this.transitionClouds.push(matchedCloud, childOnlyCloud, mergedOnlyCloud); + + this.activeAnimations.push({ + type: 'mergeTransition', + matchedCloud, childOnlyCloud, mergedOnlyCloud, + matchedStart, matchedEnd, + matchedCount: mLen, + coStart, coEnd, coCount: coLen, + moStart, moEnd, moCount: moLen, + mergedCluster: merged, + childPaths: evt.children, + startTime: performance.now(), + duration: this.mergeDuration * 1000, + onComplete: () => { + this.worldGroup.remove(matchedCloud); + this.worldGroup.remove(childOnlyCloud); + this.worldGroup.remove(mergedOnlyCloud); + matchedGeom.dispose(); matchedMat.dispose(); + coGeom.dispose(); coMat.dispose(); + moGeom.dispose(); moMat.dispose(); + this.transitionClouds = this.transitionClouds.filter( + c => c !== matchedCloud && c !== childOnlyCloud && c !== mergedOnlyCloud + ); + merged.pointCloud.visible = true; + merged.pointCloud.material.opacity = 1; + } + }); + } + + writeWorldPos(cluster, localIdx, out, offset) { + const pos = cluster.pointCloud.geometry.attributes.position; + const s = cluster.fitScale || 1; + const hp = cluster.hierarchyPosition; + out[offset] = hp.x + pos.getX(localIdx) * s; + out[offset + 1] = hp.y + pos.getY(localIdx) * s; + out[offset + 2] = hp.z + pos.getZ(localIdx) * s; + } + + writeColor(cluster, localIdx, out, offset) { + const col = cluster.pointCloud.geometry.attributes.color; + out[offset] = col.getX(localIdx); + out[offset + 1] = col.getY(localIdx); + out[offset + 2] = col.getZ(localIdx); + } + + makeTransitionMaterial(opacity) { + return new THREE.PointsMaterial({ + size: 5.0, + vertexColors: true, + sizeAttenuation: false, + transparent: true, + opacity, + depthWrite: false + }); + } + + animateFadeIn(cluster) { + if (!cluster.pointCloud) return; + cluster.pointCloud.material.opacity = 0; + cluster.pointCloud.material.transparent = true; + this.activeAnimations.push({ + type: 'fadeIn', cluster, + startTime: performance.now(), + duration: this.leafFadeDuration * 1000 + }); + } + + animateFadeOut(cluster) { + if (!cluster.pointCloud) return; + cluster.pointCloud.material.transparent = true; + this.activeAnimations.push({ + type: 'fadeOut', cluster, + startTime: performance.now(), + duration: this.leafFadeDuration * 1000, + onComplete: () => { + cluster.pointCloud.visible = false; + cluster.pointCloud.material.opacity = 1; + } + }); + } + + animateFallbackMerge(child, targetPos) { + if (!child.pointCloud || !targetPos) return; + const startPos = child.group.position.clone(); + const endPos = targetPos.clone(); + const startScale = child.group.scale.x; + this.activeAnimations.push({ + type: 'fallbackMerge', cluster: child, + startPos, endPos, startScale, + startTime: performance.now(), + duration: this.mergeDuration * 1000, + onComplete: () => { + child.pointCloud.visible = false; + child.pointCloud.material.opacity = 1; + child.group.position.copy(child.hierarchyPosition); + child.group.scale.setScalar(startScale); + } + }); + } + + update(dt) { + const now = performance.now(); + for (let i = this.activeAnimations.length - 1; i >= 0; i--) { + const a = this.activeAnimations[i]; + const t = Math.min((now - a.startTime) / a.duration, 1); + const e = this.easeInOutCubic(t); + + switch (a.type) { + case 'fadeIn': + a.cluster.pointCloud.material.opacity = e; + break; + + case 'fadeOut': + a.cluster.pointCloud.material.opacity = 1 - e; + break; + + case 'fallbackMerge': + a.cluster.group.position.lerpVectors(a.startPos, a.endPos, e); + a.cluster.group.scale.setScalar(a.startScale * (1 - e * 0.5)); + a.cluster.pointCloud.material.opacity = 1 - e * 0.85; + break; + + case 'mergeTransition': { + const mArr = a.matchedCloud.geometry.attributes.position.array; + for (let j = 0; j < a.matchedCount; j++) { + const j3 = j * 3; + mArr[j3] = a.matchedStart[j3] + (a.matchedEnd[j3] - a.matchedStart[j3]) * e; + mArr[j3 + 1] = a.matchedStart[j3 + 1] + (a.matchedEnd[j3 + 1] - a.matchedStart[j3 + 1]) * e; + mArr[j3 + 2] = a.matchedStart[j3 + 2] + (a.matchedEnd[j3 + 2] - a.matchedStart[j3 + 2]) * e; + } + a.matchedCloud.geometry.attributes.position.needsUpdate = true; + + const coArr = a.childOnlyCloud.geometry.attributes.position.array; + for (let j = 0; j < a.coCount; j++) { + const j3 = j * 3; + coArr[j3] = a.coStart[j3] + (a.coEnd[j3] - a.coStart[j3]) * e; + coArr[j3 + 1] = a.coStart[j3 + 1] + (a.coEnd[j3 + 1] - a.coStart[j3 + 1]) * e; + coArr[j3 + 2] = a.coStart[j3 + 2] + (a.coEnd[j3 + 2] - a.coStart[j3 + 2]) * e; + } + a.childOnlyCloud.geometry.attributes.position.needsUpdate = true; + a.childOnlyCloud.material.opacity = 1 - e * 0.3; + + const moArr = a.mergedOnlyCloud.geometry.attributes.position.array; + for (let j = 0; j < a.moCount; j++) { + const j3 = j * 3; + moArr[j3] = a.moStart[j3] + (a.moEnd[j3] - a.moStart[j3]) * e; + moArr[j3 + 1] = a.moStart[j3 + 1] + (a.moEnd[j3 + 1] - a.moStart[j3 + 1]) * e; + moArr[j3 + 2] = a.moStart[j3 + 2] + (a.moEnd[j3 + 2] - a.moStart[j3 + 2]) * e; + } + a.mergedOnlyCloud.geometry.attributes.position.needsUpdate = true; + a.mergedOnlyCloud.material.opacity = e; + break; + } + } + + if (t >= 1) { + if (a.onComplete) a.onComplete(); + this.activeAnimations.splice(i, 1); + } + } + } + + easeInOutCubic(t) { + return t < 0.5 ? 4 * t * t * t : 1 - Math.pow(-2 * t + 2, 3) / 2; + } +} diff --git a/visualization/hierarchy-web/js/camera-engine.js b/visualization/hierarchy-web/js/camera-engine.js new file mode 100644 index 000000000..1ccaff78e --- /dev/null +++ b/visualization/hierarchy-web/js/camera-engine.js @@ -0,0 +1,40 @@ +import * as THREE from 'three'; + +export class CameraEngine { + constructor(camera, orbitControls) { + this.camera = camera; + this.orbitControls = orbitControls; + this.flythrough = null; + } + + flyTo(position, target, duration = 1.5) { + this.flythrough = { + startPos: this.camera.position.clone(), + endPos: position.clone(), + startTarget: this.orbitControls.target.clone(), + endTarget: target.clone(), + startTime: performance.now(), + duration: duration * 1000 + }; + } + + stopFlythrough() { + this.flythrough = null; + } + + update(time) { + if (!this.flythrough) return; + + const f = this.flythrough; + const t = Math.min((performance.now() - f.startTime) / f.duration, 1); + const e = t < 0.5 ? 2 * t * t : 1 - Math.pow(-2 * t + 2, 2) / 2; + + this.camera.position.lerpVectors(f.startPos, f.endPos, e); + this.orbitControls.target.lerpVectors(f.startTarget, f.endTarget, e); + this.orbitControls.update(); + + if (t >= 1) { + this.flythrough = null; + } + } +} diff --git a/visualization/hierarchy-web/js/data-loader-vggt.js b/visualization/hierarchy-web/js/data-loader-vggt.js new file mode 100644 index 000000000..aad0b7b84 --- /dev/null +++ b/visualization/hierarchy-web/js/data-loader-vggt.js @@ -0,0 +1,580 @@ +import * as THREE from 'three'; + +export class Cluster { + constructor(path, type, childrenPaths = []) { + this.path = path; + this.type = type; // 'vggt' or 'merged' + this.childrenPaths = childrenPaths; + + this.group = new THREE.Group(); + this.pointCloud = null; + this.pointsCount = 0; + + this.slabPosition = new THREE.Vector3(); + this.originalCenter = new THREE.Vector3(); + this.radius = 0; + this.centroid = new THREE.Vector3(); + + this.rect = null; // { x, y, w, h } + + this.parent = null; + this.children = []; + } + + setPointCloud(geometry, material) { + this.pointCloud = new THREE.Points(geometry, material); + this.pointCloud.userData = { cluster: this }; + this.group.add(this.pointCloud); + + geometry.computeBoundingSphere(); + this.centroid.copy(geometry.boundingSphere.center); + this.radius = geometry.boundingSphere.radius; + this.pointsCount = geometry.attributes.position.count; + } +} + +export class VGGTDataLoader { + constructor() { + this.clusters = new Map(); + this.root = null; + this.globalCenter = new THREE.Vector3(); + this.globalRadius = 0; + this.scaleFactor = 1.0; + this.sceneRotation = new THREE.Matrix4(); + } + + async load() { + const structure = this.getStructure(); + const flatPaths = this.flattenStructure(structure); + + let loaded = 0; + const total = flatPaths.length; + + for (const item of flatPaths) { + const cluster = new Cluster(item.path, item.type, item.children); + this.clusters.set(item.path, cluster); + } + + // Link parents/children + for (const [path, cluster] of this.clusters) { + for (const childPath of cluster.childrenPaths) { + const child = this.clusters.get(childPath); + if (child) { + cluster.children.push(child); + child.parent = cluster; + } + } + } + + // Load geometry + const promises = flatPaths.map(async (item, index) => { + await this.loadPointCloud(item.path, index); + loaded++; + if (this.onProgress) this.onProgress(loaded, total); + }); + + await Promise.all(promises); + + await this.loadCameraExtrinsics(); + this.computeSceneOrientation(); + this.computeGlobalBoundsAndNormalize(); + this.computePointMatching(); + + return this.clusters; + } + + async loadPointCloud(path, colorIndex) { + try { + const fullPath = `data/gerrard-hall-vggt/results/${path}`; + const response = await fetch(`${fullPath}/points3D.txt`); + if (!response.ok) throw new Error(`Failed to fetch ${fullPath}`); + const text = await response.text(); + + const positions = []; + const colors = []; + + const lines = text.split('\n'); + for (let line of lines) { + if (line.startsWith('#') || line.trim() === '') continue; + + const parts = line.trim().split(/\s+/); + if (parts.length < 8) continue; + + const x = parseFloat(parts[1]); + const y = parseFloat(parts[2]); + const z = parseFloat(parts[3]); + const r = parseInt(parts[4]); + const g = parseInt(parts[5]); + const b = parseInt(parts[6]); + + positions.push(x, y, z); + colors.push(r / 255, g / 255, b / 255); + } + + if (positions.length > 0) { + const geometry = new THREE.BufferGeometry(); + geometry.setAttribute('position', new THREE.Float32BufferAttribute(positions, 3)); + geometry.setAttribute('color', new THREE.Float32BufferAttribute(colors, 3)); + + geometry.computeBoundingSphere(); + const originalRadius = geometry.boundingSphere.radius; + + const posAttr = geometry.attributes.position; + let sumX = 0, sumY = 0, sumZ = 0; + for (let i = 0; i < posAttr.count; i++) { + sumX += posAttr.getX(i); + sumY += posAttr.getY(i); + sumZ += posAttr.getZ(i); + } + const originalCenter = new THREE.Vector3( + sumX / posAttr.count, + sumY / posAttr.count, + sumZ / posAttr.count + ); + + const material = new THREE.PointsMaterial({ + size: 5.0, + vertexColors: true, + sizeAttenuation: false, + transparent: true, + opacity: 1.0 + }); + + const cluster = this.clusters.get(path); + if (cluster) { + cluster.setPointCloud(geometry, material); + cluster.originalCenter.copy(originalCenter); + cluster.centroid.copy(originalCenter); + cluster.radius = originalRadius; + } + } + } catch (e) { + console.warn(`Error loading ${path}:`, e); + } + } + + computeGlobalBoundsAndNormalize() { + let minX = Infinity, maxX = -Infinity; + let minY = Infinity, maxY = -Infinity; + let minZ = Infinity, maxZ = -Infinity; + let hasPoints = false; + + for (const [path, cluster] of this.clusters) { + if (cluster.pointCloud && cluster.pointCloud.geometry) { + const pos = cluster.pointCloud.geometry.attributes.position; + for (let i = 0; i < pos.count; i++) { + const x = pos.getX(i); + const y = pos.getY(i); + const z = pos.getZ(i); + minX = Math.min(minX, x); maxX = Math.max(maxX, x); + minY = Math.min(minY, y); maxY = Math.max(maxY, y); + minZ = Math.min(minZ, z); maxZ = Math.max(maxZ, z); + hasPoints = true; + } + } + } + + if (!hasPoints) { + console.warn("No points loaded, skipping normalization"); + return; + } + + const mergedCluster = this.clusters.get('merged'); + if (mergedCluster && mergedCluster.pointCloud && mergedCluster.pointCloud.geometry) { + const pos = mergedCluster.pointCloud.geometry.attributes.position; + let sumX = 0, sumY = 0, sumZ = 0; + for (let i = 0; i < pos.count; i++) { + sumX += pos.getX(i); + sumY += pos.getY(i); + sumZ += pos.getZ(i); + } + this.globalCenter.set(sumX / pos.count, sumY / pos.count, sumZ / pos.count); + } else { + this.globalCenter.set((minX + maxX) / 2, (minY + maxY) / 2, (minZ + maxZ) / 2); + } + + const sizeX = maxX - minX; + const sizeY = maxY - minY; + const sizeZ = maxZ - minZ; + this.globalRadius = Math.sqrt(sizeX*sizeX + sizeY*sizeY + sizeZ*sizeZ) / 2; + + const TARGET_SIZE = 300; + this.scaleFactor = this.globalRadius > 0 ? TARGET_SIZE / this.globalRadius : 1.0; + + for (const [path, cluster] of this.clusters) { + if (cluster.pointCloud && cluster.pointCloud.geometry) { + const geometry = cluster.pointCloud.geometry; + + geometry.translate(-this.globalCenter.x, -this.globalCenter.y, -this.globalCenter.z); + geometry.applyMatrix4(this.sceneRotation); + geometry.scale(this.scaleFactor, this.scaleFactor, this.scaleFactor); + + geometry.computeBoundingSphere(); + + // Center geometry at origin so fitScale works symmetrically + const center = geometry.boundingSphere.center.clone(); + geometry.translate(-center.x, -center.y, -center.z); + geometry.computeBoundingSphere(); + + cluster.originalCenter.copy(center); + cluster.radius = geometry.boundingSphere.radius; + + cluster.pointCloud.material.size = 5.0; + cluster.pointCloud.material.needsUpdate = true; + } + } + } + + async loadCameraExtrinsics() { + this.cameraPositions = []; + this.cameraUps = []; + this.cameraLooks = []; + + try { + const response = await fetch('data/gerrard-hall-vggt/results/merged/images.txt'); + if (!response.ok) throw new Error('Failed to fetch images.txt'); + const text = await response.text(); + const lines = text.split('\n'); + + for (let i = 0; i < lines.length; i++) { + const line = lines[i].trim(); + if (line.startsWith('#') || line === '') continue; + + const parts = line.split(/\s+/); + if (parts.length < 10) continue; + + const qw = parseFloat(parts[1]); + const qx = parseFloat(parts[2]); + const qy = parseFloat(parts[3]); + const qz = parseFloat(parts[4]); + const tx = parseFloat(parts[5]); + const ty = parseFloat(parts[6]); + const tz = parseFloat(parts[7]); + + // Rotation matrix R (world-to-camera) from quaternion + const R = [ + [1 - 2*(qy*qy + qz*qz), 2*(qx*qy - qz*qw), 2*(qx*qz + qy*qw)], + [2*(qx*qy + qz*qw), 1 - 2*(qx*qx + qz*qz), 2*(qy*qz - qx*qw)], + [2*(qx*qz - qy*qw), 2*(qy*qz + qx*qw), 1 - 2*(qx*qx + qy*qy)] + ]; + + // R^T (transpose = camera-to-world) + const RT = [ + [R[0][0], R[1][0], R[2][0]], + [R[0][1], R[1][1], R[2][1]], + [R[0][2], R[1][2], R[2][2]] + ]; + + // Camera world position: C = -R^T * t + const cx = -(RT[0][0]*tx + RT[0][1]*ty + RT[0][2]*tz); + const cy = -(RT[1][0]*tx + RT[1][1]*ty + RT[1][2]*tz); + const cz = -(RT[2][0]*tx + RT[2][1]*ty + RT[2][2]*tz); + this.cameraPositions.push(new THREE.Vector3(cx, cy, cz)); + + // Camera up in world: R^T * [0, -1, 0] (COLMAP Y points down) + const ux = -RT[0][1]; + const uy = -RT[1][1]; + const uz = -RT[2][1]; + this.cameraUps.push(new THREE.Vector3(ux, uy, uz)); + + // Camera look direction in world: R^T * [0, 0, 1] + const lx = RT[0][2]; + const ly = RT[1][2]; + const lz = RT[2][2]; + this.cameraLooks.push(new THREE.Vector3(lx, ly, lz)); + + i++; // skip POINTS2D line + } + + console.log(`Loaded ${this.cameraPositions.length} camera extrinsics from merged/images.txt`); + } catch (e) { + console.warn('Could not load camera extrinsics, using identity rotation:', e); + } + } + + computeSceneOrientation() { + if (!this.cameraPositions || this.cameraPositions.length === 0) { + console.warn('No camera data available, using identity rotation'); + return; + } + + const n = this.cameraPositions.length; + + // --- Find "up" via PCA plane fit on camera positions --- + const camCentroid = new THREE.Vector3(); + for (const p of this.cameraPositions) camCentroid.add(p); + camCentroid.divideScalar(n); + + // Build 3x3 covariance matrix + const cov = [[0,0,0],[0,0,0],[0,0,0]]; + for (const p of this.cameraPositions) { + const dx = p.x - camCentroid.x; + const dy = p.y - camCentroid.y; + const dz = p.z - camCentroid.z; + cov[0][0] += dx*dx; cov[0][1] += dx*dy; cov[0][2] += dx*dz; + cov[1][0] += dy*dx; cov[1][1] += dy*dy; cov[1][2] += dy*dz; + cov[2][0] += dz*dx; cov[2][1] += dz*dy; cov[2][2] += dz*dz; + } + + // Power iteration to find smallest eigenvector of the covariance matrix. + // Use inverse iteration: repeatedly solve (cov - sigma*I)^-1 * v to find + // the eigenvector closest to sigma=0 (smallest eigenvalue). + // Simpler approach: find largest eigenvector, deflate, find next largest, + // then the remaining is the smallest. But even simpler: use the average + // camera up vector since our analysis showed 0.9998 agreement with PCA. + const avgUp = new THREE.Vector3(); + for (const u of this.cameraUps) avgUp.add(u); + avgUp.divideScalar(n).normalize(); + + // Verify consistency: check that individual ups agree with average + let agreement = 0; + for (const u of this.cameraUps) agreement += u.dot(avgUp); + agreement /= n; + console.log(`Camera up consistency: ${agreement.toFixed(4)} (1.0 = perfect)`); + + // --- Find "front" via camera centroid → building centroid --- + const bldgCentroid = new THREE.Vector3(); + let pointCount = 0; + const mergedCluster = this.clusters.get('merged'); + if (mergedCluster && mergedCluster.pointCloud) { + const pos = mergedCluster.pointCloud.geometry.attributes.position; + for (let i = 0; i < pos.count; i++) { + bldgCentroid.x += pos.getX(i); + bldgCentroid.y += pos.getY(i); + bldgCentroid.z += pos.getZ(i); + } + bldgCentroid.divideScalar(pos.count); + pointCount = pos.count; + } + + let frontDir = new THREE.Vector3(); + if (pointCount > 0) { + frontDir.subVectors(bldgCentroid, camCentroid).normalize(); + } else { + // Fallback: use average camera look direction + for (const l of this.cameraLooks) frontDir.add(l); + frontDir.divideScalar(n).normalize(); + } + + // --- Build orthonormal scene basis --- + // scene_Y = up direction + const sceneY = avgUp.clone(); + + // scene_Z = toward viewer, orthogonalized against up + let sceneZ = frontDir.clone(); + // Remove component along sceneY to make perpendicular + sceneZ.sub(sceneY.clone().multiplyScalar(sceneZ.dot(sceneY))).normalize(); + + // scene_X = right axis + const sceneX = new THREE.Vector3().crossVectors(sceneY, sceneZ).normalize(); + + // Re-orthogonalize Z + sceneZ.crossVectors(sceneX, sceneY).normalize(); + + // The rotation matrix M has these as ROWS (transforms world → scene): + // M * v = [v·sceneX, v·sceneY, v·sceneZ] + this.sceneRotation.set( + sceneX.x, sceneX.y, sceneX.z, 0, + sceneY.x, sceneY.y, sceneY.z, 0, + sceneZ.x, sceneZ.y, sceneZ.z, 0, + 0, 0, 0, 1 + ); + + console.log(`Scene orientation computed:`); + console.log(` Up (scene Y): ${sceneY.x.toFixed(4)}, ${sceneY.y.toFixed(4)}, ${sceneY.z.toFixed(4)}`); + console.log(` Right (scene X): ${sceneX.x.toFixed(4)}, ${sceneX.y.toFixed(4)}, ${sceneX.z.toFixed(4)}`); + console.log(` Front (scene Z): ${sceneZ.x.toFixed(4)}, ${sceneZ.y.toFixed(4)}, ${sceneZ.z.toFixed(4)}`); + } + + computePointMatching() { + console.log('\n=== COMPUTING POINT MATCHING ==='); + + for (const [path, cluster] of this.clusters) { + if (cluster.type !== 'merged' || cluster.childrenPaths.length === 0) continue; + if (!cluster.pointCloud) continue; + + const mergedGeom = cluster.pointCloud.geometry; + const mergedPos = mergedGeom.attributes.position; + const mc = cluster.originalCenter; + const mCount = mergedPos.count; + + const mergedX = new Float32Array(mCount); + const mergedY = new Float32Array(mCount); + const mergedZ = new Float32Array(mCount); + for (let i = 0; i < mCount; i++) { + mergedX[i] = mergedPos.getX(i) + mc.x; + mergedY[i] = mergedPos.getY(i) + mc.y; + mergedZ[i] = mergedPos.getZ(i) + mc.z; + } + + const childEntries = []; + for (const childPath of cluster.childrenPaths) { + const child = this.clusters.get(childPath); + if (!child || !child.pointCloud) continue; + const cPos = child.pointCloud.geometry.attributes.position; + const cc = child.originalCenter; + for (let i = 0; i < cPos.count; i++) { + childEntries.push({ + childPath, + childIdx: i, + nx: cPos.getX(i) + cc.x, + ny: cPos.getY(i) + cc.y, + nz: cPos.getZ(i) + cc.z + }); + } + } + + if (childEntries.length === 0) continue; + + const candidates = []; + for (const ce of childEntries) { + let bestSq = Infinity; + let bestIdx = -1; + for (let j = 0; j < mCount; j++) { + const dx = ce.nx - mergedX[j]; + const dy = ce.ny - mergedY[j]; + const dz = ce.nz - mergedZ[j]; + const sq = dx * dx + dy * dy + dz * dz; + if (sq < bestSq) { bestSq = sq; bestIdx = j; } + } + candidates.push({ + childPath: ce.childPath, + childIdx: ce.childIdx, + mergedIdx: bestIdx, + distSq: bestSq + }); + } + + candidates.sort((a, b) => a.distSq - b.distSq); + + const p90Idx = Math.floor(candidates.length * 0.9); + const thresholdSq = candidates[p90Idx].distSq * 4; + + const mergedClaimed = new Set(); + const childMatched = new Set(); + const matchedPairs = []; + + for (const c of candidates) { + if (c.distSq > thresholdSq) continue; + if (mergedClaimed.has(c.mergedIdx)) continue; + const key = `${c.childPath}:${c.childIdx}`; + if (childMatched.has(key)) continue; + + matchedPairs.push({ + childPath: c.childPath, + childIdx: c.childIdx, + mergedIdx: c.mergedIdx + }); + mergedClaimed.add(c.mergedIdx); + childMatched.add(key); + } + + // Unmatched child points: fly to nearest merged point (many-to-one allowed) + const childOnlyPoints = []; + for (const ce of childEntries) { + const key = `${ce.childPath}:${ce.childIdx}`; + if (!childMatched.has(key)) { + let bestSq = Infinity, bestIdx = 0; + for (let j = 0; j < mCount; j++) { + const dx = ce.nx - mergedX[j]; + const dy = ce.ny - mergedY[j]; + const dz = ce.nz - mergedZ[j]; + const sq = dx * dx + dy * dy + dz * dz; + if (sq < bestSq) { bestSq = sq; bestIdx = j; } + } + childOnlyPoints.push({ + childPath: ce.childPath, + childIdx: ce.childIdx, + flyToMergedIdx: bestIdx + }); + } + } + + // Unmatched merged points: fly from nearest child point (many-to-one allowed) + const mergedOnlyIndices = []; + for (let i = 0; i < mCount; i++) { + if (!mergedClaimed.has(i)) { + const mx = mergedX[i], my = mergedY[i], mz = mergedZ[i]; + let bestSq = Infinity, bestCE = null; + for (const ce of childEntries) { + const dx = ce.nx - mx; + const dy = ce.ny - my; + const dz = ce.nz - mz; + const sq = dx * dx + dy * dy + dz * dz; + if (sq < bestSq) { bestSq = sq; bestCE = ce; } + } + mergedOnlyIndices.push({ + mergedIdx: i, + flyFromChildPath: bestCE ? bestCE.childPath : null, + flyFromChildIdx: bestCE ? bestCE.childIdx : -1 + }); + } + } + + cluster.matchData = { matchedPairs, childOnlyPoints, mergedOnlyIndices }; + const totalChild = childEntries.length; + const matchRate = totalChild > 0 ? (matchedPairs.length / totalChild * 100).toFixed(1) : 0; + console.log(`${path}: ${matchedPairs.length} matched (${matchRate}%), ${childOnlyPoints.length} child-fly, ${mergedOnlyIndices.length} merged-fly`); + } + + console.log('=== POINT MATCHING COMPLETE ===\n'); + } + + flattenStructure(structure) { + const flatPaths = []; + + const traverse = (obj, prefix = '') => { + for (const [key, value] of Object.entries(obj)) { + if (value && typeof value === 'object' && value.type) { + const path = prefix ? `${prefix}/${key}` : key; + flatPaths.push({ + path, + type: value.type, + children: value.children || [] + }); + } + + if (value && typeof value === 'object') { + const path = prefix ? `${prefix}/${key}` : key; + if (!value.type) { + traverse(value, path); + } + } + } + }; + + traverse(structure); + return flatPaths; + } + + getStructure() { + return { + 'vggt': { type: 'vggt', children: [] }, + + 'C_1': { + 'vggt': { type: 'vggt', children: [] } + }, + + 'C_2': { + 'vggt': { type: 'vggt', children: [] } + }, + + 'C_3': { + 'vggt': { type: 'vggt', children: [] } + }, + + 'C_4': { + 'vggt': { type: 'vggt', children: [] }, + 'C_4_1': { + 'vggt': { type: 'vggt', children: [] } + }, + 'C_4_2': { + 'vggt': { type: 'vggt', children: [] } + }, + 'merged': { type: 'merged', children: ['C_4/vggt', 'C_4/C_4_1/vggt', 'C_4/C_4_2/vggt'] } + }, + + 'merged': { type: 'merged', children: ['vggt', 'C_1/vggt', 'C_2/vggt', 'C_3/vggt', 'C_4/merged'] } + }; + } +} diff --git a/visualization/hierarchy-web/js/interaction-engine.js b/visualization/hierarchy-web/js/interaction-engine.js new file mode 100644 index 000000000..9a2ae7fdd --- /dev/null +++ b/visualization/hierarchy-web/js/interaction-engine.js @@ -0,0 +1,45 @@ +import * as THREE from 'three'; + +export class InteractionEngine { + constructor(camera, domElement, clusters, orbitControls) { + this.camera = camera; + this.domElement = domElement; + this.clusters = clusters; + this.orbitControls = orbitControls; + this.raycaster = new THREE.Raycaster(); + this.raycaster.params.Points.threshold = 2; + this.mouse = new THREE.Vector2(); + this.hoveredCluster = null; + + this.domElement.addEventListener('mousemove', (e) => this.onMouseMove(e)); + } + + onMouseMove(event) { + this.mouse.x = (event.clientX / window.innerWidth) * 2 - 1; + this.mouse.y = -(event.clientY / window.innerHeight) * 2 + 1; + } + + update() { + this.raycaster.setFromCamera(this.mouse, this.camera); + const pointClouds = []; + for (const cluster of this.clusters.values()) { + if (cluster.pointCloud && cluster.pointCloud.visible) { + pointClouds.push(cluster.pointCloud); + } + } + const intersects = this.raycaster.intersectObjects(pointClouds); + + if (this.hoveredCluster) { + this.hoveredCluster.pointCloud.material.size = 2.0; + this.hoveredCluster = null; + } + + if (intersects.length > 0) { + const cluster = intersects[0].object.userData.cluster; + if (cluster) { + this.hoveredCluster = cluster; + cluster.pointCloud.material.size = 3.5; + } + } + } +} diff --git a/visualization/hierarchy-web/js/layout-engine-squareness.js b/visualization/hierarchy-web/js/layout-engine-squareness.js new file mode 100644 index 000000000..160c92489 --- /dev/null +++ b/visualization/hierarchy-web/js/layout-engine-squareness.js @@ -0,0 +1,328 @@ +import * as THREE from 'three'; + +/** + * Squareness-Based Recursive Rectangle Layout Engine + * + * Only LEAF nodes get spatial positions. Merged (non-leaf) nodes + * get a mergeTargetPosition and mergeRegion used during merge animations. + */ +export class SquarenessLayoutEngine { + constructor(clusters) { + this.clusters = clusters; + this.rootCluster = clusters.get('merged'); + this.bounds = null; + this.treeNodes = []; + this.PADDING_FRAC = 0.015; + } + + compositions(n) { + const result = []; + const generate = (remaining, current) => { + if (remaining === 0) { result.push([...current]); return; } + for (let first = 1; first <= remaining; first++) { + current.push(first); + generate(remaining - first, current); + current.pop(); + } + }; + generate(n, []); + return result; + } + + squarenessForRow(W, H, n, m) { + const rho = (H * m * m) / (W * n); + return rho <= 1 ? rho : 1 / rho; + } + + bestEqualAreaPartition(W, H, n) { + if (n === 1) { + return { + squareness: Math.min(W, H) / Math.max(W, H), + mode: 'rows', groups: [1], + layout: [{ x: 0, y: 0, w: W, h: H }] + }; + } + + const allComps = this.compositions(n); + let bestS = -1, bestMode = 'rows', bestGroups = null; + + for (const ms of allComps) { + const s = Math.min(...ms.map(m => this.squarenessForRow(W, H, n, m))); + if (s > bestS) { bestS = s; bestMode = 'rows'; bestGroups = ms; } + } + for (const ms of allComps) { + const s = Math.min(...ms.map(m => this.squarenessForRow(H, W, n, m))); + if (s > bestS) { bestS = s; bestMode = 'cols'; bestGroups = ms; } + } + + const layout = this.buildTiles(W, H, bestMode, bestGroups, n); + return { squareness: bestS, mode: bestMode, groups: bestGroups, layout }; + } + + buildTiles(W, H, mode, groups, n) { + const tiles = []; + if (mode === 'rows') { + let yOff = 0; + for (const m of groups) { + const rowH = H * m / n; + const tileW = W / m; + for (let i = 0; i < m; i++) + tiles.push({ x: i * tileW, y: yOff, w: tileW, h: rowH }); + yOff += rowH; + } + } else { + let xOff = 0; + for (const m of groups) { + const colW = W * m / n; + const tileH = H / m; + for (let i = 0; i < m; i++) + tiles.push({ x: xOff, y: i * tileH, w: colW, h: tileH }); + xOff += colW; + } + } + return tiles; + } + + computeLayout() { + if (!this.rootCluster) { + console.error("No root cluster (merged) found!"); + return; + } + + console.log("\n=== COMPUTING SQUARENESS LAYOUT ==="); + + const visited = new Set(); + const buildTree = (cluster, depth = 0) => { + if (!cluster || visited.has(cluster.path)) return null; + visited.add(cluster.path); + const node = { cluster, depth, children: [] }; + this.treeNodes.push(node); + for (const child of (cluster.children || [])) { + const cn = buildTree(child, depth + 1); + if (cn) node.children.push(cn); + } + return node; + }; + const rootNode = buildTree(this.rootCluster); + + const leaves = this.treeNodes.filter(n => n.children.length === 0); + const leafCount = leaves.length; + const radii = Array.from(this.clusters.values()) + .filter(c => c.radius > 0).map(c => c.radius); + const maxRadius = radii.length > 0 ? Math.max(...radii) : 1; + + const tileSize = maxRadius * 2.2; + const totalArea = tileSize * tileSize * leafCount * 1.1; + const aspect = 16 / 9; + const ROOT_H = Math.sqrt(totalArea / aspect); + const ROOT_W = ROOT_H * aspect; + + console.log(`Leaves: ${leafCount}, maxRadius: ${maxRadius.toFixed(1)}`); + console.log(`Root rect: ${ROOT_W.toFixed(0)} x ${ROOT_H.toFixed(0)}`); + + const rootRect = { x: -ROOT_W / 2, y: -ROOT_H / 2, w: ROOT_W, h: ROOT_H }; + this.assignLeafTiles(rootNode, rootRect); + this.computeMergePositions(rootNode); + + let minX = Infinity, maxX = -Infinity, minY = Infinity, maxY = -Infinity; + + for (const node of this.treeNodes) { + const c = node.cluster; + const isLeaf = node.children.length === 0; + + if (isLeaf && c.rect) { + const cx = c.rect.x + c.rect.w / 2; + const cy = c.rect.y + c.rect.h / 2; + c.hierarchyPosition = new THREE.Vector3(cx, cy, 0); + c.group.position.copy(c.hierarchyPosition); + + if (c.radius > 0) { + const fitDim = Math.min(c.rect.w, c.rect.h) * 0.85; + c.fitScale = fitDim / (2 * c.radius); + c.group.scale.setScalar(c.fitScale); + } + + minX = Math.min(minX, c.rect.x); + maxX = Math.max(maxX, c.rect.x + c.rect.w); + minY = Math.min(minY, c.rect.y); + maxY = Math.max(maxY, c.rect.y + c.rect.h); + + } else if (!isLeaf) { + const pos = c.mergeTargetPosition; + const reg = c.mergeRegion; + if (pos) { + c.hierarchyPosition = pos.clone(); + c.group.position.copy(c.hierarchyPosition); + + if (c.radius > 0 && reg) { + const fitDim = Math.min(reg.w, reg.h) * 0.85; + c.fitScale = fitDim / (2 * c.radius); + c.group.scale.setScalar(c.fitScale); + } + } + } + } + + if (minX === Infinity) { + this.bounds = { minX: -10, maxX: 10, minY: -10, maxY: 10, width: 60, height: 60 }; + } else { + this.bounds = { + minX, maxX, minY, maxY, + width: maxX - minX + 40, + height: maxY - minY + 40 + }; + } + + for (const [path] of this.clusters) { + if (!visited.has(path)) this.clusters.get(path).group.visible = false; + } + + console.log(`Bounds: ${this.bounds.width.toFixed(0)} x ${this.bounds.height.toFixed(0)}`); + console.log("=== SQUARENESS LAYOUT COMPLETE ===\n"); + } + + countLeaves(node) { + if (!node) return 0; + if (node.children.length === 0) return 1; + let sum = 0; + for (const child of node.children) sum += this.countLeaves(child); + return sum; + } + + assignLeafTiles(node, rect) { + if (!node) return; + if (node.children.length === 0) { + node.cluster.rect = { ...rect }; + return; + } + + const children = node.children; + const n = children.length; + const weights = children.map(c => this.countLeaves(c)); + const totalWeight = weights.reduce((a, b) => a + b, 0); + const pad = Math.min(rect.w, rect.h) * this.PADDING_FRAC; + + // Generate all possible partitions of children into contiguous groups. + // For n children, there are 2^(n-1) ways to split (each gap is break or not). + const partitions = []; + const numSplits = 1 << (n - 1); + for (let mask = 0; mask < numSplits; mask++) { + const groups = []; + let start = 0; + for (let bit = 0; bit < n - 1; bit++) { + if (mask & (1 << bit)) { + groups.push({ from: start, to: bit + 1 }); + start = bit + 1; + } + } + groups.push({ from: start, to: n }); + partitions.push(groups); + } + + // Evaluate each partition in both row and column orientations. + let bestScore = -1; + let bestPartition = null; + let bestOrientation = 'rows'; + + for (const groups of partitions) { + for (const orientation of ['rows', 'cols']) { + const primaryDim = orientation === 'rows' ? rect.h : rect.w; + const crossDim = orientation === 'rows' ? rect.w : rect.h; + let worst = 1; + + for (const g of groups) { + let groupWeight = 0; + for (let i = g.from; i < g.to; i++) groupWeight += weights[i]; + const groupPrimary = primaryDim * (groupWeight / totalWeight); + + for (let i = g.from; i < g.to; i++) { + const childCross = crossDim * (weights[i] / groupWeight); + const w = (orientation === 'rows' ? childCross : groupPrimary) - pad; + const h = (orientation === 'rows' ? groupPrimary : childCross) - pad; + if (w > 0 && h > 0) { + worst = Math.min(worst, Math.min(w, h) / Math.max(w, h)); + } else { + worst = 0; + } + } + } + + if (worst > bestScore) { + bestScore = worst; + bestPartition = groups; + bestOrientation = orientation; + } + } + } + + // Apply the best partition. + let primaryOff = 0; + const primaryTotal = bestOrientation === 'rows' ? rect.h : rect.w; + const crossTotal = bestOrientation === 'rows' ? rect.w : rect.h; + + for (const g of bestPartition) { + let groupWeight = 0; + for (let i = g.from; i < g.to; i++) groupWeight += weights[i]; + const groupPrimary = primaryTotal * (groupWeight / totalWeight); + + let crossOff = 0; + for (let i = g.from; i < g.to; i++) { + const childCross = crossTotal * (weights[i] / groupWeight); + let childRect; + if (bestOrientation === 'rows') { + childRect = { + x: rect.x + crossOff + pad / 2, + y: rect.y + primaryOff + pad / 2, + w: childCross - pad, + h: groupPrimary - pad + }; + } else { + childRect = { + x: rect.x + primaryOff + pad / 2, + y: rect.y + crossOff + pad / 2, + w: groupPrimary - pad, + h: childCross - pad + }; + } + this.assignLeafTiles(children[i], childRect); + crossOff += childCross; + } + primaryOff += groupPrimary; + } + } + + computeMergePositions(node) { + if (!node || node.children.length === 0) return; + + for (const child of node.children) this.computeMergePositions(child); + + const leafRects = []; + const gatherLeaves = (n) => { + if (n.children.length === 0 && n.cluster.rect) { + leafRects.push(n.cluster.rect); + } + for (const ch of n.children) gatherLeaves(ch); + }; + gatherLeaves(node); + + if (leafRects.length === 0) return; + + let minX = Infinity, maxX = -Infinity, minY = Infinity, maxY = -Infinity; + for (const r of leafRects) { + minX = Math.min(minX, r.x); + maxX = Math.max(maxX, r.x + r.w); + minY = Math.min(minY, r.y); + maxY = Math.max(maxY, r.y + r.h); + } + + const cx = (minX + maxX) / 2; + const cy = (minY + maxY) / 2; + + node.cluster.mergeTargetPosition = new THREE.Vector3(cx, cy, 0); + node.cluster.mergeRegion = { + x: minX, y: minY, + w: maxX - minX, + h: maxY - minY + }; + } +} diff --git a/visualization/hierarchy-web/js/main-hierarchy-vggt.js b/visualization/hierarchy-web/js/main-hierarchy-vggt.js new file mode 100644 index 000000000..9f1a488f3 --- /dev/null +++ b/visualization/hierarchy-web/js/main-hierarchy-vggt.js @@ -0,0 +1,312 @@ +import * as THREE from 'three'; +import { OrbitControls } from 'three/addons/controls/OrbitControls.js'; +import { VGGTDataLoader } from './data-loader-vggt.js?v=17'; +import { SquarenessLayoutEngine } from './layout-engine-squareness.js?v=17'; +import { InteractionEngine } from './interaction-engine.js?v=4'; +import { SquarenessAnimationEngine } from './animation-engine-squareness.js?v=16'; +import { CameraEngine } from './camera-engine.js?v=4'; + +class VGGTHierarchyApp { + constructor() { + this.initThree(); + this.initUI(); + } + + initThree() { + this.container = document.getElementById('canvas-container'); + + this.scene = new THREE.Scene(); + this.scene.background = new THREE.Color(0xffffff); + + this.camera = new THREE.PerspectiveCamera(60, window.innerWidth / window.innerHeight, 0.1, 50000); + this.camera.position.set(0, 0, 200); + + this.renderer = new THREE.WebGLRenderer({ antialias: true }); + this.renderer.setPixelRatio(window.devicePixelRatio); + this.renderer.setSize(window.innerWidth, window.innerHeight); + this.renderer.outputColorSpace = THREE.SRGBColorSpace; + this.container.appendChild(this.renderer.domElement); + + this.orbitControls = new OrbitControls(this.camera, this.renderer.domElement); + this.orbitControls.enableDamping = true; + this.orbitControls.dampingFactor = 0.05; + this.orbitControls.autoRotate = false; + + const ambientLight = new THREE.AmbientLight(0xffffff, 0.9); + this.scene.add(ambientLight); + const dirLight = new THREE.DirectionalLight(0xffffff, 0.4); + dirLight.position.set(10, 10, 10); + this.scene.add(dirLight); + + this.worldGroup = new THREE.Group(); + this.scene.add(this.worldGroup); + + window.addEventListener('resize', () => { + this.camera.aspect = window.innerWidth / window.innerHeight; + this.camera.updateProjectionMatrix(); + this.renderer.setSize(window.innerWidth, window.innerHeight); + }); + } + + initUI() { + this.ui = { + loading: document.getElementById('loading'), + loadingText: document.querySelector('.loading-text'), + eventLabel: document.getElementById('event-label'), + progressBar: document.getElementById('timeline-progress'), + stats: document.getElementById('stats-display'), + prevBtn: document.getElementById('btn-prev'), + nextBtn: document.getElementById('btn-next'), + playBtn: document.getElementById('btn-play'), + resetBtn: document.getElementById('btn-reset'), + track: document.getElementById('timeline-track') + }; + + this.ui.prevBtn.addEventListener('click', () => this.step(-1)); + this.ui.nextBtn.addEventListener('click', () => this.step(1)); + this.ui.resetBtn.addEventListener('click', () => this.reset()); + this.ui.playBtn.addEventListener('click', () => this.togglePlay()); + + this.ui.track.addEventListener('click', (e) => { + if (!this.animationEngine) return; + const rect = this.ui.track.getBoundingClientRect(); + const pct = (e.clientX - rect.left) / rect.width; + const index = Math.floor(pct * this.animationEngine.mergeEvents.length); + this.jumpTo(index); + }); + + this.ui.recordBtn = document.getElementById('btn-record'); + this.ui.recordBtn.addEventListener('click', () => this.toggleRecording()); + this.mediaRecorder = null; + this.recordedChunks = []; + } + + async start() { + try { + this.dataLoader = new VGGTDataLoader(); + this.dataLoader.onProgress = (loaded, total) => { + this.ui.loadingText.textContent = `Loading VGGT Clusters... ${loaded}/${total}`; + }; + + const clusters = await this.dataLoader.load(); + + let loadedCount = 0; + for (const c of clusters.values()) { + if (c.pointCloud) loadedCount++; + } + + if (loadedCount === 0) { + throw new Error("No point clouds loaded. Check that data/ directory exists."); + } + + console.log(`Loaded ${loadedCount}/${clusters.size} clusters with point data`); + + for (const cluster of clusters.values()) { + this.worldGroup.add(cluster.group); + } + + this.layoutEngine = new SquarenessLayoutEngine(clusters); + this.layoutEngine.computeLayout(); + + this.fitCameraToLayout(); + + this.animationEngine = new SquarenessAnimationEngine(clusters, this.layoutEngine, this.worldGroup); + this.events = this.animationEngine.initTimeline(); + this.currentEventIndex = 0; + + this.interactionEngine = new InteractionEngine( + this.camera, + this.renderer.domElement, + clusters, + this.orbitControls + ); + + this.cameraEngine = new CameraEngine(this.camera, this.orbitControls); + + if (this.events.length > 0) { + this.animationEngine.applyEventInstant(0); + } + this.updateUI(); + + this.ui.loading.style.display = 'none'; + + this.animate(); + + } catch (err) { + console.error("App Start Error:", err); + this.ui.loadingText.innerHTML = `Error starting app:
${err.message}`; + } + } + + fitCameraToLayout() { + const b = this.layoutEngine.bounds; + if (!b || b.width === undefined || b.height === undefined) { + console.warn("No layout bounds, using defaults"); + this.camera.position.set(0, 0, 200); + this.orbitControls.target.set(0, 0, 0); + this.orbitControls.update(); + return; + } + + const fov = this.camera.fov; + const aspect = window.innerWidth / window.innerHeight; + + const vFovRad = THREE.MathUtils.degToRad(fov / 2); + const hFovRad = Math.atan(aspect * Math.tan(vFovRad)); + + const distForHeight = (b.height / 2) / Math.tan(vFovRad); + const distForWidth = (b.width / 2) / Math.tan(hFovRad); + + let dist = Math.max(distForHeight, distForWidth) * 1.1; + dist = Math.max(dist, 8); + + const centerY = (b.maxY + b.minY) / 2; + const centerX = (b.maxX + b.minX) / 2; + + this.camera.position.set(centerX, centerY, dist); + this.camera.lookAt(centerX, centerY, 0); + this.orbitControls.target.set(centerX, centerY, 0); + this.orbitControls.update(); + } + + step(direction) { + if (direction > 0) { + if (this.currentEventIndex < this.events.length - 1) { + this.currentEventIndex++; + this.animationEngine.playEvent(this.currentEventIndex, 1); + } + } else { + if (this.currentEventIndex > 0) { + this.animationEngine.playEvent(this.currentEventIndex, -1); + this.currentEventIndex--; + } + } + this.updateUI(); + } + + jumpTo(index) { + if (index < 0) index = 0; + if (index >= this.events.length) index = this.events.length - 1; + this.currentEventIndex = index; + this.animationEngine.applyEventInstant(index); + this.updateUI(); + } + + reset() { + this.isPlaying = false; + this.ui.playBtn.textContent = 'Play'; + this.cameraEngine.stopFlythrough(); + this.jumpTo(0); + } + + togglePlay() { + this.isPlaying = !this.isPlaying; + this.ui.playBtn.textContent = this.isPlaying ? 'Pause' : 'Play'; + + if (this.isPlaying && this.currentEventIndex >= this.events.length - 1) { + this.jumpTo(0); + } + } + + toggleRecording() { + if (this.mediaRecorder && this.mediaRecorder.state === 'recording') { + this.mediaRecorder.stop(); + this.ui.recordBtn.textContent = 'Record'; + this.ui.recordBtn.classList.remove('recording'); + return; + } + + this.recordedChunks = []; + const canvas = this.renderer.domElement; + const stream = canvas.captureStream(30); + this.mediaRecorder = new MediaRecorder(stream, { + mimeType: 'video/webm;codecs=vp9', + videoBitsPerSecond: 5000000 + }); + + this.mediaRecorder.ondataavailable = (e) => { + if (e.data.size > 0) this.recordedChunks.push(e.data); + }; + + this.mediaRecorder.onstop = () => { + const blob = new Blob(this.recordedChunks, { type: 'video/webm' }); + const url = URL.createObjectURL(blob); + const a = document.createElement('a'); + a.href = url; + const timestamp = new Date().toISOString().replace(/[:.]/g, '-').slice(0, 19); + a.download = `gerrard-hall-recording-${timestamp}.webm`; + a.click(); + URL.revokeObjectURL(url); + }; + + this.mediaRecorder.start(); + this.ui.recordBtn.textContent = 'Stop'; + this.ui.recordBtn.classList.add('recording'); + } + + updateUI() { + const count = this.events.length; + if (count === 0) return; + + const progress = (this.currentEventIndex / (count - 1)) * 100; + this.ui.progressBar.style.width = `${progress}%`; + + const event = this.events[this.currentEventIndex]; + const eventType = event.isLeaf ? 'VGGT' : 'Merge'; + this.ui.eventLabel.textContent = `Event ${this.currentEventIndex + 1}/${count}: ${eventType} — ${event.path}`; + + let visiblePoints = 0; + let visibleClusters = 0; + for (const c of this.dataLoader.clusters.values()) { + if (c.pointCloud && c.pointCloud.visible) { + visibleClusters++; + visiblePoints += c.pointsCount; + } + } + if (this.animationEngine && this.animationEngine.transitionClouds.length > 0) { + for (const tc of this.animationEngine.transitionClouds) { + if (tc.visible && tc.geometry && tc.geometry.attributes.position) { + visiblePoints += tc.geometry.attributes.position.count; + visibleClusters++; + } + } + } + this.ui.stats.textContent = `Clusters: ${visibleClusters} | Points: ${visiblePoints.toLocaleString()}`; + } + + animate() { + requestAnimationFrame(() => this.animate()); + + const time = performance.now() / 1000; + const dt = 0.016; + + if (this.isPlaying) { + if (!this.lastStepTime) this.lastStepTime = time; + const hasActiveAnims = this.animationEngine && this.animationEngine.activeAnimations.length > 0; + if (!hasActiveAnims && time - this.lastStepTime > 1.0) { + if (this.currentEventIndex < this.events.length - 1) { + this.step(1); + this.lastStepTime = time; + } else { + this.togglePlay(); + } + } + } else { + this.lastStepTime = 0; + } + + if (this.animationEngine) { + const hadAnimations = this.animationEngine.activeAnimations.length > 0; + this.animationEngine.update(dt); + if (hadAnimations) this.updateUI(); + } + if (this.cameraEngine) this.cameraEngine.update(time); + + this.orbitControls.update(); + this.renderer.render(this.scene, this.camera); + } +} + +const app = new VGGTHierarchyApp(); +window.app = app; +app.start(); diff --git a/visualization/hierarchy-web/recordings/.gitkeep b/visualization/hierarchy-web/recordings/.gitkeep new file mode 100644 index 000000000..e69de29bb