Use linear memcpy for contiguous buffer copies (fixes init failure on meshes > ~1M elements on ROCm/HIP) - #35
Open
ron-ytsma wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CuMesh::init()copies vertex/face tensors withcudaMemcpy2Dwheredpitch == spitch == width(12-byte rows). On ROCm/HIP (tested: gfx1151,ROCm 7.x, hipified via torch's build pipeline),
hipMemcpy2Dfails withhipErrorInvalidValueonce the row count exceeds ~1M — i.e. anyreal-world TRELLIS.2 mesh. The failed call also leaves the HIP context
with a sticky error, so the crash surfaces later in unrelated torch ops,
which makes this very hard to diagnose downstream (we chased it through
the whole TRELLIS.2 decode path before isolating it here).
Minimal repro (ROCm):
<200b>
python import torch, cumesh n = 2_000_000 v = torch.rand(n, 3, device='cuda', dtype=torch.float32) f = torch.randint(0, n, (2*n, 3), device='cuda', dtype=torch.int32) m = cumesh.CuMesh() m.init(v, f) # [CuMesh] CUDA error: io.hip:124 invalid argument <200b>(n=100_000 works; n=2_000_000 fails. Independent of HSA_ENABLE_SDMA.)
Fix
Both copies in
init()are fully contiguous (pitch == width on bothsides), so
cudaMemcpy2Dis semantically a plain linear copy — replacewith
cudaMemcpyofrows * widthbytes. The genuinely-strided branchin
buffer_to_tensor()(only taken whensizeof(T) != dst_bytes) iskept as
cudaMemcpy2Dbut chunked to 65536 rows per call to stay underthe limit.
No behavior change on CUDA: the linear copy is byte-identical for the
contiguous case (and avoids any pitched-copy overhead).
Tested
init OK at 5M vertices / 10M faces; full fill_holes / simplify chains run
on 3M-face TRELLIS.2 meshes.