Planning from Point Clouds over Continuous Actions for Multi-object Rearrangement

CoRL 2025 (Oral Presentation - top 5.7%)

Kallol Saha^*1, Amber Li^*1, Angela Rodriguez-Izquierdo^*2, Lifan Yu¹, Ben Eisner¹, Maxim Likhachev¹, David Held^1
*Equal Contribution
¹Robotics Institute, Carnegie Mellon University         ²Princeton University

We present Search over Point Cloud Object Transformations (SPOT), a method that solves long-horizon planning tasks such as multi-object rearrangement, by searching for a sequence of transformations from an initial scene point cloud to a goal-satisfying point cloud. SPOT uses guided A* search, expanding nodes by sampling actions from learned suggesters that operate on partially observed point clouds, thus eliminating the need to discretize actions or object relationships.

---

This is the official repository of SPOT. If you find our work useful, please consider citing our paper:

@InProceedings{saha2025spot,
  title = 	 {Planning from Point Clouds over Continuous Actions for Multi-object Rearrangement},
  author =       {Saha, Kallol and Li, Amber and Rodriguez-Izquierdo, Angela and Yu, Lifan and Eisner, Ben and Likhachev, Maxim and Held, David},
  booktitle = 	 {Proceedings of The 9th Conference on Robot Learning},
  year = 	 {2025},
  volume = 	 {305},
  pdf = 	 {https://raw.githubusercontent.com/mlresearch/v305/main/assets/saha25a/saha25a.pdf},
  url = 	 {https://proceedings.mlr.press/v305/saha25a.html},
}

If you find any bugs in the code, or have any questions, feel free to raise an issue.

Table of Contents

• Installation
  • Clone the repository and submodules
  • Graph Visualization
  • Install SPOT
• Quick Demo
• Running our Method
  • 1. Collecting Data
  • 2. Training Suggesters
    • Training the placement suggester
    • Training the object suggester
    • Training the Model Deviation Estimator (MDE)
  • 3. Planning
  • 4. Execution
    • Simulation Execution
    • Real World Execution

Installation

Clone the repository and submodules:

git clone --recursive https://github.com/kallol-saha/SPOT.git
cd SPOT

Graph Visualization:

We need graphviz for visualizing graphs:

sudo apt-get install graphviz

If you cannot use sudo or do not want to visualize graphs, disable the global flag: (This is already set to FALSE by default)

# In files that use graph visualization (e.g., `spot/evaluation.py`, `scripts/real_world/plan_real_world.py`)
VISUALIZE_GRAPH = False  # set to True to enable graph visualization

Install:

Create a conda environment and activate it:

conda create -n spot python=3.9
conda activate spot

Make sure you have CUDA 11.7 installed (required by pytorch3d, torch cluster and torch geometric):

conda install -c nvidia/label/cuda-11.7.0 cuda-toolkit

Use the bash script below to install the remaining dependencies:

chmod +x install_spot.sh
sh install_spot.sh

Quick Demo:

Loads pre-trained models (placement suggester, object suggester, MDE), creates an A* planner, generates and evaluates a plan.

python demo.py

Running our Method:

1. We collect human demonstration data in both simulation and real world environments.
2. We train 3 learned modules from human demonstration data: (a) Placement suggester, (b) Object suggester, and (c) Model Deviation Estimator (MDE)
3. Given a test dataset of initial point clouds, we compute plans (a sequence of transformations applied to objects) using guided A* search.
4. To benchmark our method, we execute the plans found in simulation and real world environments.

1. Collecting Data:

For simulation data collection, we move objects with the keyboard and record the final transformations.

python scripts/collect_keyboard_demos.py

For real world data collection, we extract 2D Tracks, from which we extract 2D transformations. See this repo

2. Training Suggesters:

You can train each of the suggesters from scratch, or you can skip this step and use the pre-trained weights downloaded under assets and run planning and execution directly.

Training the placement suggester

Trains an equivariant multi-modal transformer network (TAXPose-D) that suggests object placement poses given point cloud observations to learn placement distributions from demonstration data.

python scripts/training/train_placement_suggester.py

Training the object suggester

Trains a PointNet++ classifier model that determines which objects should be moved next during planning:

python scripts/training/train_object_suggester.py

Training the Model Deviation Estimator (MDE)

Trains a PointNet++ model that predicts how much the environment state differs from expected after executing actions. Supports both classification and regression modes, with datasets for blocks and table bussing environments.

python scripts/training/train_mde.py

3. Planning:

Evaluates the planning performance of SPOT across test datasets. Loads trained models, runs guided A* search-based planning on point clouds, and saves metrics including success rates, planning times, and node expansion counts.

python scripts/planning/benchmark_plans.py

4. Execution:

Simulation Execution:

Executes the plans generated by the planner in a PyBullet simulation environment. Applies the planned object transformations sequentially using motion planning, records execution success/failure, and saves execution metrics to evaluate plan feasibility.

python scripts/planning/benchmark_sim_execution.py

Real World Execution:

We use a Franka Panda for executing SPOT in a real world environment.
scripts/real_world/plan_real_world.py: Generates real world plans
scripts/real_world/get_plans_from_goals.py: Selects the best plan