feat: lazy blob-backed shape store for STEP and IFC imports#235
Open
Krande wants to merge 86 commits into
Open
feat: lazy blob-backed shape store for STEP and IFC imports#235Krande wants to merge 86 commits into
Krande wants to merge 86 commits into
Conversation
…metry Large STEP/IFC imports materialised as ada.geom trees cost ~10x their serialized size in resident memory (Munin crane: 2.6 GB settled vs 275 MB of NGEOM blobs). ShapeStore keeps each solid as one compact blob (NGEOM buffers retained zero-copy as they arrive from adacpp; Python-built geometry pickled losslessly incl. bool_operations) and hydrates the Geometry tree on demand behind a WeakValueDictionary, mirroring the FEM array-substrate design. ShapeProxy subclasses Shape, holds only (store, index), and exposes the raw blob for adacpp fast paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…irst auto reader - Config cad.lazy_shape_store (default on, ADA_CAD_LAZY_SHAPE_STORE=false to opt out) + cad.shape_store_compress: _read_step_streaming stores each solid as its NGEOM blob (native path, zero hydration at import) or pickled ada.geom tree (pure-Python path) and mints ShapeProxy objects. - reader='auto' now probes the native adacpp parser first (matching iter_from_step), with a first-solid hydrate probe and pure-Python fallback; StepReader.NATIVE + from_step Literal accept 'native' explicitly. - native_stream_read_step_blobs: per-solid (blob, meta) without hydration; native_stream_read_step reimplemented on top. - ConnectedFaceSet becomes a first-class B-rep root: solid_geom() accepts it, the OCC builder builds it, and NGEOM hydration promotes topologically closed face-sets back to ClosedShell (edge-pairing check, value-keyed). - AP242 stream writer shares EDGE_CURVEs by value signature instead of object identity, so NGEOM-hydrated trees (which decode one object per reference) re-emit watertight solids identical to Python-reader trees. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- AdaCppBackend.tessellate_stream_buffer: tessellate a pre-encoded NGEOM buffer (a ShapeStore blob) directly — no hydration, no re-serialization; tessellate_stream now routes through it. - BatchTessellator: ShapeProxy solids tessellate straight from their stored blob when the stream kernel is selected; the bare-curve sniff skips proxies (it would hydrate the whole tree, and stored solids are never bare curves). - IFC writer: ShapeProxy accepted alongside Shape for parametric export; bare ConnectedFaceSet roots map to the ClosedShell face-set emit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
import_ifc_shape stores natively-read product geometry as one compact pickled blob per product (bool_operations, half-space operands and parametric profiles round-trip exactly) on a per-IfcStore ShapeStore and mints ShapeProxy objects; kernel-fallback products keep eager Shapes. Gated by the same cad.lazy_shape_store config (default on). Crane numbers for the STEP side of the store (native reader, 7291 solids): eager 2723/2556 MB peak/settled 96 s -> lazy 709/543 MB 56 s; all 7291 blobs hydrate cleanly and RSS returns to the blob floor after dropping the trees. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ilter The exact-type filter excluded lazy proxies from by_type=Shape queries (topology from_assembly found no solids). A proxy normalises to its public type; Shape subclasses like PrimBox stay excluded as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Geometry.bool_operations were dropped by every stream-tessellation caller (the wrapper was stripped before serialization), so IFC clipping cuts and API booleans only evaluated on the OCC build path. Now: - serialize root() accepts core.Geometry wrappers and folds bool_operations into a nested BOOLEAN_RESULT (tag 52) chain — base as first operand, ops in order — which adacpp already decodes and evaluates via Manifold. - HalfSpaceSolid operands are lowered at encode time to a finite box on the material side of the plane, sized by the base solid's loose bbox — the exact lowering adacpp's native STEP/IFC readers apply (mk_halfspace), so no new NGEOM tag and no adacpp change is needed; the neutral path stays OCC-free (pure ada.geom bbox math). - BatchTessellator passes the wrapper through to tessellate_stream. Native-read CSG STEP roots (BOOLEAN_RESULT in an adacpp blob) still have no Python-side decode: the auto reader's hydrate-probe falls back to the pure- Python reader for such files, which parses BooleanResult natively. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Removes the per-instance __dict__ from every curve/surface/solid/placement/ boolean dataclass — the dominant per-object overhead when a B-rep model is hydrated (millions of instances). core.Geometry stays un-slotted: it is the weakref-cache value in the lazy ShapeStore (weakref_slot needs py>=3.11) and there is only one per solid. Two NGEOM serializer tests attached weights_data ad hoc to the non-rational B-spline classes; they now construct the Rational subclasses (real field). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
B-rep IFC products the Python-native readers can't resolve previously fell to the eager IfcOpenShell OCC kernel (unpicklable transient bodies, whole- model materialisation). They now try adacpp's dep-free IfcNgeomStream first: one guid-keyed scan per import, buffers retained zero-copy in the per-IfcStore ShapeStore, single-instance placement restored from the stream's composed world matrix. Color/props/spatial hierarchy still come from ifcopenshell (the stream's v1 carries geometry+guid+name+placement only); kernel fallback unchanged for products the native resolver skips, and adacpp builds without IfcNgeomStream degrade gracefully. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Audit regression: non-promoted native NGEOM roots hydrate as bare ConnectedFaceSet with FaceSurface members; the adacpp backend's build dispatch only knew the ClosedShell/OpenShell + AdvancedFace spellings and raised 'not yet ported'. ConnectedFaceSet sews like the other shells; FaceSurface is AdvancedFace's structurally-identical sibling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion LRU - SAT wire bodies round-tripped through IFC are bare-curve shapes; the proxy curve-sniff skip forced them down solid_geom() and they vanished from scenes (parity ifc 1->0). ShapeRecord now records curve-ness at store time and the tessellator asks the record — line rendering restored with zero hydration. - ~40% conversion slowdown: consumers read .geom 3-5x per call and each temporary died between property evaluations, so the weak cache re-decoded the blob per access. ShapeStore adds a small strong LRU window (hydration_cache_size=16, ~5 MB) over the weak cache; hydrate-all still returns to the blob floor (eviction test included). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
promote_closed_shell (edge-pairing closedness check) cost ~17% of native_stream_read_step on the crane, and the streaming exporters (obj/stl/step emit) that iterate it don't care about shell closedness. Promotion now happens only where it matters: lazy-store hydration (already there) and the eager from_step Assembly wrap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🚀 Profiling Results (Top 20 most expensive calls)
|
…filer When the admin profile_conversions setting is on, the fork child gets ADACPP_STEP_PROFILE=1 alongside cProfile: adacpp prints [STEPPROF] phase wall times, RSS at phase boundaries, VmHWM peak, per-solid stats and parallelism/IO pressure to stderr, which the captured job Log keeps — the C++ sibling of the .prof artefact. Scoped per job like the other env overrides; nothing is baked into the container. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When profile_conversions is on, the worker parses the adacpp profiler's [STEPPROF-JSON] summaries out of the captured child log into convert_meta.cpp_profile, and the Metrics tab renders them next to the Python profile: per-pipeline phase wall/RSS breakdown with share bars, kernel-exact peak (VmHWM), per-solid stats, achieved parallelism, disk IO pressure and per-thread utilisation. Torn/interleaved lines are skipped; profiling off leaves no trace in meta or output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The audit log panel gains a state dropdown (queued / running / done / error) next to the action/target filters. Server-side: the /admin/audit route exposes a status param wired to list_audit's existing statuses filter, so it composes with keyset pagination instead of filtering one page client-side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ete-key + shift-arrow selection Corpus tab tree view gains the storage panel's organize toolkit via a shared FileTreeView mutations layer: inline rename of files/folders, client-side pending folders (New folder / New subfolder), kebab menus (download, move-to-folder picker, upload-here, delete with cascades), and drag-and-drop moves of files and whole folder subtrees with a move-to-root strip. Drag payloads embed the source scope in a distinct MIME so drags straying between panels/scopes are ignored, not mis-moved. Checkbox multi-select feeds a bulk Move/Copy/Delete toolbar and multi-key drags. Both panels: "Upload files" now prompts for the destination (existing folder / new path / top level, default top level) via FolderPickerModal allowRoot; keyboard lists support shift+Arrow selection extension and Delete-key deletes (selection first, else the focused file/folder) with confirm dialogs listing exactly which keys go (previewKeyList). Corpus files can be server-side copied into the caller's personal scope (Garage CopyObject, keys preserved, existing keys skipped) per file or per selection. InlineNameInput extracted from StorageBrowser into components/common so both panels share it. Read-only FileTreeView callers are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anel PositionedMenu and FolderPickerModal portal to document.body at z-50, but the admin Rnd panel host is a fixed z-[60] overlay — so the corpus tab's kebab/context menus and its move/upload destination prompts rendered behind the panel. Raise both to z-[70]. (The copy-from-scope modal was unaffected: it renders inline inside the panel's stacking context.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…escription
Both storage panels now show an in-panel spinner status line while a
batch file operation runs — previously a drag-drop move of many files
gave no feedback until the listing refreshed. Corpus tab: bar under the
button row covering moves, folder moves, bulk/folder deletes, and
copy-to-personal; storage panel: line under the header for drag-drop,
picker, and folder moves. Multi-key moves/copies are chunked (8 keys
per request) purely so the counter ticks between requests — every chunk
is still a server-side S3 rename/copy (CopyObject on Garage); no file
bytes pass through the browser. A busy ref rejects overlapping batches,
which would race on server-side collision checks.
Corpus name and description are now editable inline from the corpus
files header (pencil toggle) via a new PATCH /admin/corpora/{slug}
endpoint + db.update_corpus. The slug stays immutable — it is baked
into the storage prefix and scope URLs, so renaming it would orphan the
bucket bytes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…try dialog Corpus upload batches now pre-check destination keys against the listing and skip files already in the corpus instead of overwriting — same semantics as the copy flows; the outcome line reports uploaded/skipped counts. Failures no longer collapse into one error string: they open a dialog listing each failed file with its reason and Retry / Close actions. Retry re-attempts only the failed files (the File objects are held, no re-pick needed), and the skip pre-check makes retrying a partially-succeeded upload a safe no-op. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The corpus tree (and other FileTreeView selections) only supported per-row checkbox toggles and shift+arrow — shift+click did nothing, reading as a regression next to the storage panel's range select. Add the same anchor semantics: shift+click selects the visible file range from the last toggled row (adds, never removes); keyboard and checkbox paths keep the anchor in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mple set Ten files, five root causes, all fixed at the source: - Inch/foot units (scale 0.0254): the reader compared the file's length unit against the mm/m enum and raised for conversion-based units. store.py now compares raw scales and converts through the existing generic convert_file_length_units (6 files / 30 cells). - IfcTriangulatedFaceSet: imported fine but no backend had a build for it, so shapes silently tessellated to an empty scene. It IS a mesh — emit it directly via a new direct-mesh path in BatchTessellator (plus solid_geom() allowlist entries for TriangulatedFaceSet + FacetedBrep). - Type-library files (geometry only on an IfcTypeProduct RepresentationMap, no placed products — the texture samples): new instance-less type import, gated to files where no placed product carried geometry so normal models don't grow phantom shapes. - IfcRectangleProfileDef: parametric_profile_to_arbitrary now emits the centred rectangle outline (2 files, glb/obj/stl), and the AP242 stream writer converts parametric profiles through the same seam instead of assuming .outer_curve (ifc->step). - IfcTrimmedCurve parameter trims: the reader now preserves the IfcLine vector magnitude (parameterization scale) and normalizes conic trim angles to radians via the file's plane-angle unit (the degrees/radians sample pair now produce identical meshes). Both backends evaluate parameter trims: adacpp encodes line/circle natively and samples ellipse arcs; OCC builds trimmed line/circle/ellipse edges with the explicit XDirection frame. The 2D-placement ellipse reader crash is fixed the same way circle() handles it. - Products with no spatial container (IFC4x3 alignment-hosted signals on IfcLinearPlacement) attach to the assembly root instead of crashing on parent.Name. Regression tests over the (public) buildingSMART samples in files/ifc_files/bs_samples/, green on both CAD backends; full core suites pass (pyocc 1129, adacpp 1144). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An auto-validate run's total previously covered only the conversion grid; the parity cells were appended (extend_audit_run_total) when the run finished and the poller dispatched the validation pass, so the displayed total grew mid-run. The total now includes the parity cells from initial dispatch via a validate_total reservation column (migration 021): - set_audit_run_total reserves the parity count inside total, so the counter-bump finish check keeps the run 'running' through the gap between the last conversion cell and the validation dispatch - claim_audit_run_for_auto_validate claims reserved runs once their conversion cells alone have landed (still 'running'); legacy rows with validate_total=0 keep the finished+extend behaviour - consume_audit_run_validation_reserve swaps the reservation for the actual parity cell count at dispatch (total only moves on scope drift; zero actual finishes the run) — failures release the reserve so a run can't hang on cells that will never be enqueued - one storage listing feeds both the conversion grid and the parity reservation so the two counts can't disagree Manual Validate on a finished run still extends the total as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…osses) Audit run 63 (first run on the fixed worker image) confirmed 51 of the 54 conversion fixes and left 16 cells: 3 ifc re-exports the type-import fix unmasked, and 13 cross-format parity losses the newly-appended validation phase caught for the first time. ifc re-export (texture trio): - IfcStore.get_context creates the missing standard subcontext under the model context for imported files that carry none (the buildingSMART type-library samples have only bare '3D'/'2D' contexts) - TriangulatedFaceSet/FacetedBrep write parametrically (1:1 IfcTriangulatedFaceSet emit) instead of falling into the tessellation fallback, which needs a kernel build these kinds didn't have parity losses: - TriangulatedFaceSet builds on both backends (triangle faces sewn to a shell, mirroring PolygonalFaceSet) so mesh bodies reach STEP - GradientCurve directrix encodes as the shared alignment-evaluator's sampled polyline on both backends; OCC MakeSolid failure on an open sweep degrades to the swept shell instead of dropping the body - bare-curve shapes (SAT wire bodies) route through active_backend().build in the STEP export instead of a pythonocc-only wire builder, with build_wire/edge-record support on the adacpp backend - bare IfcClosedShell/IfcOpenShell representation items import again (write_shapes emits imported shells this way) - the DOM Genie-XML writer emits BeamRevolve/BeamSweep as straight chord beams like the streaming writer, instead of dropping them - the parity STEP counter falls back to a reload count when the native stream sees 0 roots (wireframe-only outputs are invisible to it) 8 new cross-format parity regression tests over the previously-failing sources; both core suites green (adacpp 1151, pyocc 1136; the one failure is the pre-existing plotly-to-file/kaleido issue, verified on a clean tree). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pty-model parity phantom Audit run 64 (image sha-b29eb92) cleared all 16 prior failures but flipped 12 parity cells to an inverted mismatch (source=0, step=1). Three root causes: - the parity element counter's reload fallback counted a zero-vertex phantom: a geometry-less STEP product materializes one empty Shape whose scene entry has no vertices. visualized_element_count skips renderably-empty entries, and tessellate_edges no longer passes an edge-less compound to discretize_edge (TopoDS_Edge assert) - wire-only STEP files read back as zero roots: GEOMETRIC_CURVE_SET was a referenced-entity builder but not a stream-reader ROOT, so adapy's own wireframe exports (SAT wire bodies) were invisible on re-read. It is now a yieldable root and a first-class curve body: CURVE_GEOM_TUPLE membership, native line discretization (incl. line-basis TrimmedCurve), per-element GL_LINES emit, OCC compound-of-wires build, and adacpp edge-record encode - the 12 files were passing vacuously (0=0=0): their products import as ZERO objects because a typed import failure (e.g. IfcBeam with a tessellated body and no material association) dropped the product entirely. Typed Beam/Plate/Wall imports now fall back to the generic Shape import on any failure, and the IfcTriangulatedFaceSet reader handles the optional Normals attribute. The tessellated I-beam sample now imports, renders and round-trips (parity 1=1=1) instead of silently showing an empty scene New regression test covers the empty-model zero count on every leg and the wireframe STEP round-trip. Both core suites green (adacpp 1152, pyocc 1137; the single failure is the pre-existing plotly-to-file/kaleido issue). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cProfile of the JacketHybrid parity cell (the audit's slowest Python cell, ~99s on the worker) showed round_array at ~28s cumulative: every call built a fresh np.vectorize(roundoff) to round a 3-vector, 637k times through compute_orientation_vec in FEM-shell->Plate conversion and the IFC/XML plate emitters. A plain per-element loop keeps roundoff's exact Decimal HALF_EVEN values (swapping in np.round shifted borderline vectors enough to drop 59 of 104k shells in plate conversion — rejected) and cuts FEM->concept conversion on that model from 59s to 46.5s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mbled scene cProfile of the hullskin parity cell showed ~44% of its wall time inside assembly_element_count: to_trimesh_scene tessellates the whole model at viewer quality and then assembles a full trimesh scene (trimesh objects, materials, normals) three times per cell — once for the source baseline and once per reloaded format leg — only for the entries to be counted. The count now consumes BatchTessellator.batch_tessellate directly over the same object set tessellate_part feeds it (physical objects with pipes as segments, plus welds): meshes_to_trimesh turns exactly one MeshStore into one scene entry, so counting non-point, non-empty stores is the same metric with the scene-assembly third of the cost removed. Exclusions unchanged: point clouds (the empty-scene placeholder) and zero-vertex stores. Measured on the two slowest audit parity cells, counts identical to the production baselines: hullskin.xml 130s -> 80s (5410 on every leg), JacketHybrid.FEM 86s local vs 98.6s worker (104188 on every leg). The parity regression suite (incl. the wire-STEP and empty-model edge cases that flow through this counter) is green on both backends. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
convert_shell_elem_to_plates ran the full orientation/projection chain per element (normal_to_points_in_plane -> calc_yvec -> compute_orientation_vec x2 -> 2D projection -> winding -> back-transform): on a 100k-shell model that meant 1.5M Direction / 1.3M Point constructions, 2.9M weakref-cache ops and 2M Decimal roundings — ~2/3 of FEM->concept wall time. The math now runs once over (m, k, 3) arrays: - vector_transforms.shell_orientations_bulk replicates the scalar chain array-wise in the same floating-point operation order, Decimal-rounding via round_array semantics deduplicated over unique orientation rows (co-planar plate patches share frames), and escapes degenerate rows (duplicate corners, near-parallel edges, sub-1e-9 yvec) to the scalar path - CurvePoly2d.from_fem_shells_batch consumes it: batched matmul projection, the verbatim is_clockwise winding rule (its closing term is NOT the wrapped shoelace edge — replicated bit for bit), batched back-transform; only the output objects (interned Points, LineSegments, Nodes) are built per element - convert_part_shell_elements_to_plates gathers entries in element order with the exact scalar guards/warnings, coplanarity split, tri-branch material quirk and try/except, then batches per polygon arity Gate: all 109,949 JacketHybrid plates BITWISE-IDENTICAL to the scalar chain (points2d/3d, orientation vectors, segment indices, thickness, material). Conversion 39.0s -> 11.9s on the plates alone; create_objects_from_fem 59s (session start) -> 15.4s; the Jacket parity audit cell 98.6s (worker baseline) -> 59.2s locally with identical counts. Streaming exporters keep the per-element scalar path (iter_objects_from_fem builds detached plates one at a time by design). Core suites green on both backends; FEM suite green in the fem env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…explosion) IfcSectionedSolidHorizontal (a profile swept along an alignment directrix, its cross-sections placed at distances along it) had no native reader, so it fell back to the IfcOpenShell OCC kernel, which tessellated the swept solid into thousands of loose faces (sectioned-solid-horizontal.ifc: 4802 SHELL_BASED_SURFACE_MODEL) — a faceted explosion that also can't reach the kernel-free stream path. For a CONSTANT cross-section (all sections equal) it is a fixed-reference sweep over the directrix range [first, last] cross-section distance. The reader evaluates it natively (no OCC) to a triangulated swept shell via the alignment evaluator: range-restricted directrix sampling + fixed-reference frames (profile x->lateral, y->up) + side quads + fan end caps, wound outward by signed volume. Emitted as a TriangulatedFaceSet, so it renders on the stream kernel and STEP-round-trips as ONE surface model. Varying cross-sections raise NotImplementedError (kernel fallback). Verified vs the ifcopenshell oracle: bbox exact ([300,-22.26,148.52]..[599.88,5, 149.7]); from_ifc yields 8 objects (was 7 with the solid skipped / exploded); STEP round-trip = 1 solid + 7 curves (was 4802 faces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IfcMappedItem (an instancing wrapper: a MappingSource representation reused via a MappingTarget transform) had no native reader — the single most common corpus OCC-kernel fallback (75 products across 10 files: rebar, basins, CSG, mapped shapes). It now reads natively (no IfcOpenShell kernel): - geom_reader.mapped_item: unwrap the single mapped representation item, read it natively (recurse into import_geometry_from_ifc_geom — the wrapped extrusions / swept disks / CSG / face sets are already supported), and apply the mapped-item 4x4 via ifcopenshell's pure-Python get_mappeditem_transformation (no kernel). - _transform_geometry: identity is a no-op (the common case; ObjectPlacement does the placing); a rigid (optionally uniform-scale) transform moves point-set geoms' coordinates / an IfcSweptDiskSolid's directrix + radius. Non-rigid (shear / non-uniform scale), multi-item mapped representations, and multi-item product bodies raise NotImplementedError -> kernel fallback (no instance dropping). - base.py: add SweptDiskSolid to Shape.solid_geom's accepted types — the OCC + NGEOM builders already exist, it was just never rendered (pipes/rods, rebar directrix). - read_shapes: a product with >1 Body geometries now keeps the kernel (which builds all items) instead of silently taking geometries[0]. Verified vs the ifcopenshell oracle: reinforcing-stirrup / basins / bath-csg read native and render (stirrup 0 -> 2596 tris); reinforcing-assembly's 34 mapped rebar place correctly along the directrix. Non-uniform (mapped-shape-with-transformation) and multi-item-mapped-rep (linear-placement) degrade gracefully to the kernel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
import_ifc_shape only applied the product's ObjectPlacement when it had a PlacementRelTo chain, silently dropping ABSOLUTE placements — which can still carry a non-identity rotation. A concrete IfcBeam that falls through to the shape importer (reinforcing-assembly) has an absolute placement rotating its local-Z extrusion onto world-Y; the missing placement rendered it along Z (vertical) instead of Y. get_local_placement returns the full world 4x4 for relative AND absolute placements, and native parametric geometry is always in local coords, so apply it whenever ObjectPlacement is present. Now the assembly bbox matches the ifcopenshell oracle exactly ([-0.1,0,-0.4]..[0.1,5,0]). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IfcFaceBasedSurfaceModel (a set of IfcConnectedFaceSet, each a set of IfcFace) was the last true corpus OCC-kernel fallback. It now reads natively: - surfaces.py: connected_face_set + face_based_surface_model readers build the FaceBasedSurfaceModel -> ConnectedFaceSet -> Face -> FaceBound -> PolyLoop hierarchy; dispatch branches added (IfcConnectedFaceSet after its IfcClosed/OpenShell subtypes so those keep their specific readers). - base.py: FaceBasedSurfaceModel added to Shape.solid_geom, and the placement baker extended to POINT-BASED / FACE-SET geometries (transforms vertices for Triangulated/Polygonal face sets and the face-set/surface-model hierarchy) — the long-noted follow-up. So surface-model.ifc, with its +1 m x ObjectPlacement, lands in world coordinates (bbox now matches the ifcopenshell oracle exactly). Consistency fix exposed by the above: the beam shell (geom_beams.py) put bare FaceBounds in ConnectedFaceSet.cfs_faces, violating the list[Face] contract that the IFC reader honours. Wrap them in a Face so the shared face-set builders (OCC make_shell_from_connected_face_set_geom + adacpp build) handle both producers; the OCC FaceBasedSurfaceModel builder is rewritten to reuse that shared path (the old one assumed cfs_face.bound and errored on Face), and adacpp build reads Face.bounds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Some exporters/servers ship a gzip'd model under a plain .ifc name (the buildingSMART beam-standard-case sample is one), so ifcopenshell.open fails with 'Unable to parse IFC SPF header' and the file imports as nothing. Add open_ifc_file: detect the gzip magic (0x1f 0x8b) and load the inflated SPF text via from_string, else open normally. Wired into both IfcStore open paths (__post_init__ + ifc_obj_from_ifc_file) and reader_utils.open_ifc, so from_ifc reads it like any other model (beam-standard-case: 18 beams). Fixture stored gzip'd (marked binary). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gallery mode gains a walk-type dropdown. Besides the existing "files" walk (cycles the scope's loadable files), two new walks cycle the geoms already in the scene, selecting + framing (fit object) each in turn: - "geoms": every draw-range in the scene, ordered by "scene" (load order) or "density" (triangles per surface area — heaviest first). - "tree": the same geoms in model-tree hierarchy order. An "Isolate" toggle optionally hides everything except the current geom during the walk (hide-all-except helper built on hideBatchDrawRange + unhideAllRanges, since hides are additive-only). Each step drives the same selection + Object-Info path a click does (highlight, name, server metadata) and frames the camera via centerViewOnSelection. Arrow keys cycle prev/next for whichever walk is active. New collapsible "Mesh" section in the selected-object info panel: per- selection tessellation stats (triangles, vertices, surface area, bbox volume, density) computed client-side from the batched geometry buffer (shared computeRangeStats also backs the density ordering). Walk type / order / isolate preferences persist in the gallery store. Frontend typechecks (tsc --noEmit clean) and builds (vite) clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The top-level walk is now just Files / Geoms; "tree" (model-tree hierarchy order) becomes a third Geoms order alongside scene and density, so there's one geom slideshow with a single order selector rather than two overlapping top-level walk types. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…x expansion Two native-IFC profile/curve bugs surfaced by corpus review: - IfcRoundedRectangleProfileDef is a SUBTYPE of IfcRectangleProfileDef, so the rectangle dispatch branch swallowed it and dropped RoundingRadius — the bath-csg void read as a sharp rectangle (sharp interior corners). Add a preceding branch that builds a rectangle with 4 quarter-circle corner fillets as an ArbitraryProfileDef (verified: void corner verts lie exactly on R). - IfcIndexedPolyCurve's IfcLineIndex is a POLYLINE through all its points (N points = N-1 edges), but the reader kept only the first two points, collapsing multi-point runs. An I-section's flange outline (IfcLineIndex with 6 points) degenerated to a single edge → the extruded I-beam came out a ~half-width box. Expand each run to consecutive edges (beam-extruded-solid now builds the full IPE200 matching the ifcopenshell oracle). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors the Object Info name-copy: clicking the file path (files walk) or geom name (geoms walk) in the gallery HUD copies it, with a brief '✓ copied' flash.
The ACIS entity header `$owner -1 -1 $attrib` has two bare `-1` integers that are numeric, so the plane-surface / straight-curve parsers' plain float-filter swallowed them and shifted origin/normal/u_direction by two — planes got a garbage frame (a flat plate then tessellated to a single triangle on the NGEOM libtess2 path). Add _numeric_after_header (skip to the 2nd $-reference) and use it in both parsers. (The adacpp NGEOM tessellator also needed a fix so the same plate's mixed straight/trimmed-B-spline boundary loop doesn't self-intersect.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Right-click an audit cell -> "Rerun cell ↻" re-runs just that one cell in place instead of re-dispatching the whole 900+ matrix. It enqueues a single force-rebuild conversion against the run's own scope/pool, re-points the cell's audit row at the new job, undoes the counter its prior status bumped, and reopens the run (db.reset_audit_cell_for_rerun); the worker's normal completion path re-increments and re-finishes the run. Also: a run's displayed total runtime is now the SUM of every cell's own duration_ms (get_audit_run / list_audit_runs expose cells_duration_ms), not wall clock — parallel workers compress wall clock below the real compute cost, and a single-cell rerun would otherwise inflate it with the idle gap since the first run. The sum recomputes server-side whenever a cell's row changes, so a rerun's new timing shows immediately. Live-Postgres tests cover the counter/reopen/runtime transitions; full rest suite green (251). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…un form Overview list gets a sticky "Runtime" toggle switching every run's shown time between Σ cells (sum of cell runtimes — real compute cost, immune to worker parallelism and single-cell re-runs) and wall (active wall clock — time actually waited). Both are relevant; the choice persists in localStorage. The new-run form is now collapsible on mobile via a "New audit run ▾/▸" header (always shown on md+), and auto-collapses when a run is opened so the run's grid gets the small screen. No-op visually on desktop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e gap reset_audit_cell_for_rerun reopened the run (finished_at=NULL) but didn't fold the gap since the original finish into idle_ms, so when the re-run cell landed and re-finished the run, wall clock swallowed the entire days-long gap. Fold that gap into idle_ms on reopen (same as extend_audit_run_total) so wall clock grows only by the re-run's own active time. Sum-of-cells runtime already did the right thing (it replaces just that cell's duration). Live-Postgres test asserts a ~1h backdated finish is folded into idle_ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The plain-Face branch of the closed/open-shell builder used only bounds[0] (the outer loop) and dropped every inner bound, so a planar face with an opening — e.g. basin-faceted-brep.ifc's rim, whose IfcFaceOuterBound carries an IfcFaceBound hole for the mouth — built as a solid disk, capping the basin. Build all bounds via make_planar_face_from_bounds, subtracting inner loops as holes; the hole wire is reversed only when it winds like the outer (sources are inconsistent), decided from the Newell normal. Verified vs the ifcopenshell oracle: surface area 0.665 (capped) -> 0.492 (open), exactly matching. NOTE: this fixes the OCC build path. The prod libtess2/NGEOM path still caps (the FacetedBrep isn't serialized to NGEOM, and the adacpp shell build drops inner bounds) — tracked separately for an adacpp-side fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two per-cell quality flags, surfaced as a far-right "flags" column in the audit
run cell grid (aggregated per source, mirroring the PerformanceTab "streaming"
pill):
- occ_fallback: the OCC-free NGEOM/libtess2 stream tessellation silently fell
back to OCC. Counted at the single _log_tess_fallback choke point (only reached
when a stream pipeline was selected, so every hit is a genuine fallback), read
out + reset per conversion.
- distorted_tris: heavily distorted (degenerate/extreme-sliver) triangles in the
rendered mesh — the shape a displaced/collapsed vertex makes ("tris out of
place"). Vectorized aspect-ratio scan on the raw triangles as they stream
through tessellate_part/scene_from_object (before meshopt encoding, so no
decode), sampled above 300k tris to stay sub-ms.
Both are carried out of the convert subprocess via marker lines
([TESSFALLBACK-JSON]/[MESHHEALTH-JSON]) and folded into audit_log.convert_meta by
_attach_cpp_profiles, so they add negligible time and need no new DB column.
convert_meta is exposed on the AuditRunJob type and rendered as pills.
Scope note: the pure-adacpp STEP->GLB native stream tessellates in C++ and is not
yet scanned; the PerformanceTab aggregate pill is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…plane + id-memo pin) The OCC-free libtess2/NGEOM path couldn't render an IfcFacetedBrep / plain planar Face at all — no _dispatch branch and Face carries no surface — so it fell back to OCC (which capped the basin rim's opening). Now: - _dispatch serializes a FacetedBrep via its outer shell. - face_surface infers a plane (Newell normal) for a plain Face, encoding all bounds so inner-bound openings cut real holes under libtess2's odd-winding. The subtle bug that made this produce "tris out of place": surface() memoizes by id(), but each inferred plane is transient and GC'd right after serializing — the next face's plane reused the freed id(), collided in the memo, and got the PREVIOUS plane's record. Faces then tessellated onto the wrong plane, displaced by growing amounts (worse the more faces), silently dropping ~20% of surface area on a real shell. Pin every inferred plane for the encoder's lifetime. basin-faceted-brep.ifc now renders natively at area 0.49167 = the ifcopenshell oracle (was 0.665 capped via OCC / 0.399 displaced): open basin, hole cut, zero OCC fallback. Regression test builds a 120-face shell with GC churn. cadit green on the adacpp backend (359). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…it panel On mobile the gallery HUD is a full-width bottom bar and the ambient "audit sweep in progress" toast sat on top of it. Now they stack: GalleryControls publishes its live mobile-bar height (ResizeObserver; 0 when the bar is display:none on desktop) into the gallery store, and the viewer toast stack lifts its bottom to sit above the bar (audit toast on top). Desktop is unaffected (bar height 0). Also: the Audit Runs panel gets a "● toast on / ◌ toast off" toggle (persisted) so operators can hide the ambient toast entirely; the toast reads the flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switching scope disposes the current scope's batched meshes, but the geom walk held live references to them (in geomEntries + the selection). Focusing/framing a freed mesh — or a synchronous collectGeomEntries throwing inside an effect during the transition — could crash the whole viewer. - focusGeomEntry now guards: skip undefined entries and any mesh no longer attached to the live scene (meshIsLive), and re-check after the async name lookup before centerViewOnSelection reads geometry. - rebuildGeoms wraps collectGeomEntries in try/catch so a mid-transition scene can't throw out of an effect (which would unmount the app). - GalleryControls resets the walk (endGeomWalk + clear entries/index/selection) on scope change, so nothing references the old scope's disposed meshes. Pod logs showed no server error — the crash was purely client-side. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2D arc cross) _sample_arc did np.cross on a profile ArcLine's 2D points — np.cross of two 2D vectors is a scalar, so the circumcenter's np.cross(vec, scalar) raised "At least one array has zero dimension". Any parametric section with fillets (I/T/U/... — 16-segment outline, 4 fillet arcs) hit it, so its ExtrudedAreaSolid tessellated empty and fell back to OCC. Lift the arc's points to 3D (z=0) first. This was the #2 OCC-fallback cause in the audit (~38 fallbacks across the beam files — beam-standard-case, beam-parametric-cross-section, beam-varying-*, curve-parameters). beam-standard-case now renders natively (fallback 0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IfcPolygonalFaceSet had no _dispatch branch, so it serialized empty and fell back to OCC. Encode it as a CONNECTED_FACE_SET of planar polygon faces (each n-gon's 1-based indices -> a PolyLoop; face_surface infers the plane). ~24 audit fallbacks (alignment/linear-placement models) now render on the stream kernel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inferred-plane encoder (87d3caa) projected each n-gon onto a best-fit plane, which flattens the non-planar polygonal faces of swept/faceted geometry (alignment signals etc.) — area dropped ~15% vs the ifcopenshell oracle and ~90% of triangles came out as slivers. That's worse than the OCC fallback, which renders these correctly. Keep the fallback for now; a correct native path needs 3D fan/ear triangulation (keeping the original vertices), tracked separately. The ExtrudedAreaSolid arc fix (the larger fallback cause) stands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… fallback) IfcPolygonalFaceSet had no NGEOM encoder, so it fell back to OCC (~24 audit fallbacks, alignment/linear-placement models). It's already a mesh, so emit it directly: _direct_triangulated_meshstore now fan-triangulates each n-gon in 3D, keeping the ORIGINAL vertices — and _tessellate_geom_via_stream tries that direct path before serializing (libtess2 can't take a face set, and a per-face plane inference would flatten non-planar faces). Verified vs OCC: identical geometry (area 22.845, same triangles) — my earlier "native is worse" read was wrong; the high sliver fraction is the source's thin faceted strips, which OCC produces identically. Now zero OCC fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… fallback) IfcSweptDiskSolid (rebar/pipe) had no NGEOM encoder, so it fell back to OCC (~35 audit fallbacks, reinforcing-assembly etc.). Encode it like a FixedReferenceSweptAreaSolid: a circular/annular profile FACE swept along the directrix. The directrix is sampled and given a rotation-minimising (parallel-transport) frame per station here, so ANY directrix works (polyline, arc, composite) — not just the alignment GradientCurves the existing frame sampler handles. reinforcing-assembly now renders natively (fallback 0) at area 7.58 vs the ifcopenshell oracle's 7.63 (99.3%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IfcAdvancedBrep wasn't in the geom_reader dispatch, so products using it (basin advanced-brep, alignment signals) imported as raw OCC bodies with no ada.geom and fell back to OCC at tessellation. Dispatch it to closed_shell(Outer) — an outer ClosedShell of IfcAdvancedFaces (analytic / B-spline surfaces), serialized like any closed shell. Fixes 3 audit raw-OCC fallbacks (basin + 2 in the signal model); fallback now 0, geometry within B-spline-tessellation tolerance of the oracle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…level transforms)
Two IfcMappedItem forms still fell back to OCC (raw-OCC body, no ada.geom):
* a non-rigid (scale/shear) transform on a solid — can't be baked into an
analytic ada.geom type; and
* a product that instances ONE mapping source N times (multiple Body items) —
dropped by the multi-item kernel fallback.
Handle both via Geometry.transforms (per-instance 4x4s applied to the tessellated
mesh, mirroring the STEP assembly-instance path):
* mapped_item: on a non-bakeable transform, carry the 4x4 on Geometry.transforms
instead of raising.
* mapped_instance_group + read_shapes: fold shared-source mapped instances into
one Geometry with every instance transform.
* batch_tessellate (_emit_with_geom_transforms): bake each transform into the
vertex positions (inverse-transpose into normals for non-uniform scale), one
MeshStore per instance.
mapped-shape-with-transformation and mapped-shape-with-multiple-items now render
natively (fallback 0) at area/bbox identical to the ifcopenshell oracle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two more audit OCC fallbacks (trimmed-curve-parameters, surface-model): * ArbitraryClosedProfileDef with an IfcCompositeCurve outer (trimmed conic / line segments) had no NGEOM path — _loop_points_3d couldn't walk a CompositeCurve. Add _sample_trimmed_curve (conic arc via radians param or projected Cartesian trim; line segment honouring sense_agreement) + _composite_curve_loop_points, routed through _loop_points_3d. curve-parameters-in-degrees/-radians now render natively at area 33.458 vs the ifcopenshell oracle's 33.467. * IfcFaceBasedSurfaceModel of plain polyloop faces serialized empty: _any_face gated on hasattr(f, "face_surface"), which a plain IfcFace lacks. Route any face carrying `bounds` to face_surface (it already infers a plane when no surface is present). surface-model now renders natively at area 10.000 = oracle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a "distorted tris" geom-walk order that visits only geoms with a crows-nest spike (a triangle shooting out past the geometry), worst-first, with edges forced on — the tool for finding tessellation spikes. The metric (frontend meshStats + backend accumulate_mesh_distortion, kept in sync) is redefined from aspect-only to OUTLIER-VERTEX based: a spike is a thin triangle touching a vertex whose distance from the mesh's robust (median) centroid exceeds 4x the median vertex distance. This is what separates a real spike from benign geometry the old aspect>60 test over-flagged — a deep thin extrusion's side slivers or a coarse curved surface are thin and reach across the bbox, but their vertices sit ON the body. Verified: synthetic spike flagged, clean grid 0; on real files the false positives collapse — signal 0.906->0.001, curve-params 0.88->0, basin 0.093->0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The selection inspector's "Mesh" tessellation-stats block was a separate always-on section. Fold it inside the Properties expanded body (so it collapses with Properties) and gate it on a new showMeshStats option (DisplayOptions → "Mesh stats in Properties", default on). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (boiler crows-nest) scene_from_step_stream tessellated the libtess2 path at a fixed angular ceiling with NO model_scale, so a large assembly over-tessellated every small pipe/torus into a slivery "crows nest" (boiler 415583 asm_22: a single torus refined 61 -> 24105 tris; the solid hit 170k tris / 54% thin slivers vs OCC's 13.7k / 10% on the SAME NGEOM blob — a libtess2-path issue, not the source geometry). native_step_to_glb already estimates + passes model_scale (adaptive coarsening, default on); the streaming path didn't. Estimate it once (estimate_step_model_scale on the source, unit-scale corrected), expose via env to the spawn-pool workers, and pass it into tessellate_stream. Features small vs the model bbox diagonal relax their angular ceiling; significant features keep the fine tessellation. Gated by ADA_STREAM_TESS_ADAPTIVE (default on, matching the native path). Verified: boiler model_scale estimate 15364 -> r=30 tori coarsened; worst solid 56718 -> ~9000 tris (~6x fewer slivers). NOTE: a residual ~35% thin fraction remains even at OCC-comparable density — libtess2's midpoint-split refinement makes more slivers than OCC's mesher; that structural mesh-quality gap is tracked separately. Corpus-wide density change — validate with a fresh audit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
👋 Hi there! I have checked your PR and found the following: PR Review:I found no pr-related issues.
Python Review:I found some python-related issues: Python Linting results:
Python Packaging results:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Large CAD imports materialised as
ada.geomobject trees cost ~10x their serialized size in resident memory (millions of small dataclass instances, each with a__dict__). This PR keeps each imported solid as one compact blob in a sharedShapeStoreand mintsShapeProxyobjects that hydrate the geometry tree on demand — mirroring the FEM array-substrate design (ada.api.mesh).Measured on a 778 MB / 7291-solid STEP assembly (
from_step, native reader):shape_store_compressAll solids hydrate cleanly on demand and RSS returns to the blob floor when hydrated trees are dropped (weakref cache + bounded strong LRU window for back-to-back access).
Design
ada.api.shapes.ShapeStore: per-import store. NGEOM buffers from the native readers are retained zero-copy as they arrive and double as the tessellation/export fast path (tessellate_stream_buffer— no hydrate, no re-serialize). Python-built geometry (IFC readers, API shapes) stores as lossless pickle blobs, since the NGEOM solid encoders are lossy lowerings (parametric profiles -> arbitrary outlines).ShapeProxy(Shape)holds(store, index);.geomhydrates on demand,.geom =pins,ngeom_blob()feeds adacpp directly. Bare-curve shapes (wire bodies) are flagged on the record so the tessellator's line-rendering sniff never hydrates.cad.lazy_shape_store(ADA_CAD_LAZY_SHAPE_STORE=falseopts out);cad.shape_store_compressfor zlib-at-rest.reader="auto"now probes the native adacpp parser first (first-solid hydrate probe, pure-Python fallback, OCC last);StepReader.NATIVEexposed. IFC import is a three-tier hybrid: Python-parametric (pickle blob) -> nativeIfcNgeomStreamblob (B-reps the Python readers can't resolve; previously eager OCC kernel bodies) -> OCC kernel.Correctness fixes the work surfaced
Geometry.bool_operationsfold intoBOOLEAN_RESULTchains at serialization, withHalfSpaceSolidoperands lowered to finite bbox-sized boxes (the same lowering adacpp's readers use) — previously cuts were silently dropped by every stream-tessellation caller.ConnectedFaceSet(the native reader's B-rep root form) became first-class:solid_geom(), both backend builders, IFC export, and a topological closedness promotion back toClosedShell(edge-pairing with value keys — coordinate pairs collide on circle half-arcs).EDGE_CURVEs by value signature instead of object identity, so NGEOM-hydrated trees (one object per edge reference) re-emit watertight solids.slots=Trueacross allada.geomdataclasses (removes a__dict__per instance;core.Geometrystays un-slotted as the weakref-cache value).Verification
Companion adacpp PR: zero-copy stream buffers +
IfcNgeomStream.🤖 Generated with Claude Code