Skip to content

Add reverse AD for llvm.intr.memcpy#2841

Open
xys-syx wants to merge 9 commits into
mainfrom
memcpy
Open

Add reverse AD for llvm.intr.memcpy#2841
xys-syx wants to merge 9 commits into
mainfrom
memcpy

Conversation

@xys-syx

@xys-syx xys-syx commented May 28, 2026

Copy link
Copy Markdown
Collaborator

llvm.intr.memcpy has no registry in reverse AD, which cause "could not compute the adjoint for this operation "llvm.intr.memcpy"".

Reverse rule used here:

  d_src[i] += d_dst[i]
  d_dst[i] = 0

@Pangoraw Pangoraw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the rule looks good, we will need to include scftocf and cftollvm lowering passes in pipelines.

Type ptrTy = cp.getDst().getType();

auto forOp = scf::ForOp::create(builder, loc, c0, n, c1);
OpBuilder body(forOp.getBody()->getTerminator());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to use the provided builder with setInsertionPoint as it can have hooks

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Pangoraw thanks, I have modified

we will need to include scftocf and cftollvm lowering passes in pipelines

we have covert-polygeist-to-llvm in the downstream, I guess it may could work for it
https://github.com/EnzymeAD/Enzyme-JAX/blob/5db2b680308c38613ed6d5d01ed7d9d9a1353a65/src/enzyme_ad/jax/Passes/ConvertPolygeistToLLVM.cpp#L4573

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes perfect in this case

@xys-syx xys-syx requested a review from Pangoraw June 4, 2026 09:46
return t;
if (Type t = walk(cp.getSrc()))
return t;
return Float64Type::get(cp.getContext());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is wrong, we should throw an error if things can't be deduced. This is also somewhat hacky and we should use type analysis

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I will fix it

@xys-syx xys-syx Jun 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wsmoses I have fixed it, but only for the case where the inferred element type is a scalar int or float and using upstream MLIR for inferring types. The memcpy ops in RSBench that copy arrays of struct<(f64, f64)> and array<10 x f64> need to handle it recursively, I plan to open a new PR for it.

@xys-syx xys-syx requested a review from wsmoses June 8, 2026 05:20
@vimarsh6739

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db457091c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Value byteSz =
LLVM::ConstantOp::create(builder, loc, len.getType(),
builder.getIntegerAttr(len.getType(), bytes));
Value nInt = LLVM::SDivOp::create(builder, loc, len, byteSz);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Divide memcpy length as an unsigned byte count

llvm.intr.memcpy's length is a byte count/size_t, so using signed division makes large i32/i64 lengths with the high bit set become negative; after the index_cast, the generated scf.for can execute zero iterations and leave d_dst unpropagated/unzeroed. Use unsigned division for the element count here.

Useful? React with 👍 / 👎.

Comment on lines +413 to +414
Value gSrc = LLVM::GEPOp::create(builder, loc, ptrTy, elemTy, dSrc,
ArrayRef<LLVM::GEPArg>{iv});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the source shadow pointer address space

When a memcpy copies between different LLVM pointer address spaces, dSrc has the source pointer type, but this GEP is forced to return ptrTy from the destination pointer. That emits the reverse source load/store in the wrong address space (or produces invalid LLVM dialect IR) for valid llvm.intr.memcpy calls with differing src/dst address spaces; use the source pointer type for the source shadow GEP.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants