Isaac ROS Image Segmentation

NVIDIA-accelerated, deep learned semantic image segmentation

Overview

Isaac ROS Image Segmentation contains ROS packages for semantic image segmentation.

These packages provide methods for classification of an input image at the pixel level by running GPU-accelerated inference on a DNN model. Each pixel of the input image is predicted to belong to a set of defined classes. The output prediction can be used by perception functions to understand where each class is spatially in a 2D image or fuse with a corresponding depth location in a 3D scene.

Package	Model Architecture	Description
Isaac ROS U-NET	U-NET	Convolutional network popular for biomedical imaging segmentation models
Isaac ROS Segformer	Segformer	Transformer-based network that works well for objects of varying scale
Isaac ROS Segment Anything	Segment Anything	Segments any object in an image when given a prompt as to which one
Isaac ROS Segment Anything2	Segment Anything2	Segments and tracks any object in a video stream when given a prompt as to which one

Input images may need to be cropped and resized to maintain the aspect ratio and match the input resolution expected by the DNN model; image resolution may be reduced to improve DNN inference performance, which typically scales directly with the number of pixels in the image.

Image segmentation provides more information and uses more compute than object detection to produce classifications per pixel, whereas object detection classifies a simpler bounding box rectangle in image coordinates. Object detection is used to know if, and where spatially in a 2D image, the object exists. On the other hand, image segmentation is used to know which pixels belong to the class. One application is using the segmentation result, and fusing it with the corresponding depth information in order to know an object location in a 3D scene.

ROS 2 Native rosidl::Buffer Acceleration

This package uses rosidl::Buffer, a feature built into ROS 2 Lyrical, to avoid unnecessary copies of large payloads between CPU and accelerator memory. The CUDA buffer backend builds on this native ROS 2 feature to provide CUDA memory storage and transport. Most applications can use standard ROS messages and conversion packages without depending directly on a buffer backend. See rosidl::Buffer and Buffer Backends for details.

Performance

Sample Graph	Input Size	AGX Thor T5000	AGX Thor T4000	AGX Orin	Orin Nano Super 8GB	DGX Spark	x86_64 w/ RTX 5090	x86_64 w/ RTX 5070
SAM Image Segmentation Graph Full SAM	720p	5.14 fps	2.22 fps	2.22 fps	–	2.22 fps	25.0 fps	8.40 fps
SAM Image Segmentation Graph Mobile SAM	720p	20.8 fps	14.6 fps	8.40 fps	4.80 fps	14.6 fps	109 fps 10 ms @ 30Hz	57.9 fps 18 ms @ 30Hz
TensorRT Graph PeopleSemSegNet	544p	478 fps 3.8 ms @ 30Hz	352 fps 4.4 ms @ 30Hz	315 fps 4.7 ms @ 30Hz	156 fps 8.0 ms @ 30Hz	298 fps 2.9 ms @ 30Hz	1160 fps 1.1 ms @ 30Hz	935 fps 1.6 ms @ 30Hz

---

Documentation

Please visit the Isaac ROS Documentation to learn how to use this repository.

---

Packages

• isaac_ros_segformer
  • Quickstart
  • Try More Examples
  • Troubleshooting
  • API
• isaac_ros_segment_anything
  • Quickstart
  • Try More Examples
  • Troubleshooting
  • API
• isaac_ros_segment_anything2
  • Quickstart
  • Troubleshooting
  • API
• isaac_ros_segment_anything2_interfaces
• isaac_ros_unet
  • Quickstart
  • Try More Examples
  • Troubleshooting
  • API

Latest

Update 2026-09-21: Migrated the image segmentation nodes from NITROS to rosidl::Buffer with the CUDA buffer backend, and fixed a per-frame GPU buffer leak in SAM2 live-camera pipelines