Fix RVM inference on GPU execution providers - #923
Conversation
Use the original ONNX model for RVM instead of the pre-optimized .ort variant, which is CPU-EP-targeted and crashes GPU execution providers such as CUDA. Also let ONNX Runtime allocate output tensors during inference and copy the results back into the pre-allocated CPU buffers with bounds checking. Pre-binding CPU output tensors can misbehave on GPU execution providers for stateful multi-output models like RVM. Relates to royshil#759 Signed-off-by: Alex Burneikis <alexburneikis@gmail.com>
Not acceptable at all. Closing. Please open an issue instead of PR. |
|
We accept Gen-AI based PRs only if we can assume the author has ability to verify their code safely. This is the meaning of the policy on CONTRIBUTING.md. |
|
No problem, hope this code can still be used for reference / resolving the issue. 👍 |
|
Thanks. We plan to remove the RVM model because it is too large and this problem will be removed. |
Oh really? I've found this model to be the most useful, and personally, it's the only one I use, (with GPU, using my patch). Is this decision final? Is there a better suggested option? I appreciate the development effort/cost required to maintain this sort of thing. |
|
@burneikis The computational load of RVM is too large for many users and they have reported issues frequently. The large footprint of RVM is related to its inference cost. Of course, I know the quality of RVM matte is quite good. Therefore, I'm considering generating fine mask from coarse matte like the approach implemented in LBRL. If you want us to keep RVM within our plugin, please tell us why you need to use RVM and what is the key difference from every other model. We are always open but need evidences from users because we don't have telemetry in our plugin. Thank you for asking us about the current situation! |
I will test this and put together a better explanation / reasoning and get back to you 👍 |
|
I did a quick test running the models side-by-side with my webcam to best showcase my preference. RVM GPU vs CPU:
I also emulated the LBRL pipeline to test the coarse-matte idea (MediaPipe 256x144, fast guided filter, temporal smoothing, motion gate). 
As you can see, the RVM is much more accurate / less transparent-spill IMO, the key difference is the sharp/hard line 
So my request is to keep RVM, but only offer it when a GPU execution provider is active. |
|
I agree with keeping RVM for large-sized portrait. The missing key from our plugin is clarifying the situation for which models are suited. Are you good with this conclusion? |
Yeah, the main thing is that the model is slow (especially without GPU), I would only recommend for GPU use. |
|
I don't believe we can recommend RVM because GPU on many systems doesn't have enough resources for this model. |
Oh I see, so even for average GPU it's still to much? |
|
Such an intelligent detection is so great for users. However, it is hell for developers because it enlarges our responsibilities to resolve errors on every specific equipment. So, my answer is no. |
|
Fair. Perhaps just a disclaimer on the Model Selection, or in this project's README. |
|
My point of view is to add a model guide we have. I'd like to tell users that the only recommended model is MediaPipe and they can choose another model like RVM at their own based on the knowledge of models. |
2 participants
Fix RVM inference on GPU execution providers
Preface: This change was written with claude fable, I am not a C++ Systems developer. I have manually tested this change. I hope to get this issue fixed, and maybe this helps. I will not be offended if it's rejected.
To be clear, I have read contributing.md I understand to a very basic level whats happening here.
I am keeping this a draft while I try and understand this fully.
Summary
Fixes #759 (RVM / GPU inference producing a single frame then freezing or crashing).
Two changes:
.onnxmodel for RVM instead of the pre-optimized.with_runtime_opt.ortvariant, which is serialized with CPU-EP-specificoptimizations and breaks GPU execution providers (CUDA, TensorRT, etc.).
runNetworkInference()and copyresults back into the pre-allocated CPU buffers with bounds checking.
Pre-binding CPU output tensors misbehaves on GPU EPs for stateful
multi-output models like RVM. (Same root cause Fix GPU inference freezing after first frame #760 attempted; this goes
further by also replacing the CPU-targeted
.ortmodel.)Notes for reviewers:
tensor allocation + copy per frame (negligible). Output shape mismatches are
now a logged warning with truncation instead of an ORT error.
happens at load instead of being pre-baked); inference speed is unchanged.
.ortfiles.required_operators_and_types.with_runtime_opt.configmay need regeneratingsince the CI ORT build uses
--include_ops_by_config(it is not a--minimal_build, so loading ONNX-format models is supported).Test Plan
Automated
custom reduced-ops ORT build loading the
.onnxmodel..onnx.licenseadded, old.ortand license removed).
Manual - GPU (the fix)
On a machine with an NVIDIA GPU (Linux CUDA build, and Windows GPU build if
available):
to CUDA: background is removed continuously (not frozen after 1 frame),
no crash, no ORT errors in the OBS log.
switch.
(watch RSS / GPU memory).
Manual - CPU
remain stable across frames, confirming recurrent state passing works).
measurable regression from the added copy).
expected (graph optimization now happens at load); confirm it is
acceptable (< ~1 s).
Regression - other models
The inference change affects all models, so spot-check each on CPU:
(segmentation models) produce correct masks.
produce correct output.
rvm_mobilenetv3_fp32.onnxand no stale
rvm_mobilenetv3_fp32.with_runtime_opt.ort.Pull Request Checklist
Please read our latest CONTRIBUTING.md.