Skip to content

Use linear memcpy for contiguous buffer copies (fixes init failure on meshes > ~1M elements on ROCm/HIP) - #35

Open
ron-ytsma wants to merge 1 commit into
JeffreyXiang:mainfrom
ron-ytsma:fix/large-mesh-memcpy2d
Open

Use linear memcpy for contiguous buffer copies (fixes init failure on meshes > ~1M elements on ROCm/HIP)#35
ron-ytsma wants to merge 1 commit into
JeffreyXiang:mainfrom
ron-ytsma:fix/large-mesh-memcpy2d

Conversation

@ron-ytsma

Copy link
Copy Markdown

Problem

CuMesh::init() copies vertex/face tensors with cudaMemcpy2D where
dpitch == spitch == width (12-byte rows). On ROCm/HIP (tested: gfx1151,
ROCm 7.x, hipified via torch's build pipeline), hipMemcpy2D fails with
hipErrorInvalidValue once the row count exceeds ~1M — i.e. any
real-world TRELLIS.2 mesh. The failed call also leaves the HIP context
with a sticky error, so the crash surfaces later in unrelated torch ops,
which makes this very hard to diagnose downstream (we chased it through
the whole TRELLIS.2 decode path before isolating it here).

Minimal repro (ROCm):

<200b>python import torch, cumesh n = 2_000_000 v = torch.rand(n, 3, device='cuda', dtype=torch.float32) f = torch.randint(0, n, (2*n, 3), device='cuda', dtype=torch.int32) m = cumesh.CuMesh() m.init(v, f) # [CuMesh] CUDA error: io.hip:124 invalid argument <200b>

(n=100_000 works; n=2_000_000 fails. Independent of HSA_ENABLE_SDMA.)

Fix

Both copies in init() are fully contiguous (pitch == width on both
sides), so cudaMemcpy2D is semantically a plain linear copy — replace
with cudaMemcpy of rows * width bytes. The genuinely-strided branch
in buffer_to_tensor() (only taken when sizeof(T) != dst_bytes) is
kept as cudaMemcpy2D but chunked to 65536 rows per call to stay under
the limit.

No behavior change on CUDA: the linear copy is byte-identical for the
contiguous case (and avoids any pitched-copy overhead).

Tested

  • gfx1151 (AMD Strix Halo / Radeon 8060S), ROCm 7.x, torch 2.12 nightlies:
    init OK at 5M vertices / 10M faces; full fill_holes / simplify chains run
    on 3M-face TRELLIS.2 meshes.
  • The fix unblocked the complete TRELLIS.2 image→3D pipeline on AMD hardware.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant