Forensic debugging platform for autonomous robot systems. Replay missions, perform causal analysis, detect hidden objects, fuse multispectral sensors, and reconstruct what really happened—from passive replay to intelligent agent debugging.
Robotics teams waste 2-16 hours debugging a single mission failure—jumping between rosbags, logs, dashboards, and manually reconstructing causality.
Old tools answer "where/what" questions:
- Where is the robot now?
- What sensor data was captured?
PyRoboReplay 2.0 answers "why" and "what if" questions:
- Why did the robot fail? (Root cause analysis + causal graphs)
- What should have been detected? (Retrospective DINO + SAM)
- What was actually there? (RGB + thermal fusion forensics)
- What changed in the environment? (Temporal knowledge + terrain intelligence)
- Will this happen again? (Predictive modeling + pattern detection)
- How do we prevent it? (Recommendations + sensor fusion analysis)
Result: Debug 10x faster, fix failures before they happen, understand reality gaps at scale.
Comprehensive detection of perception mismatches between simulation and reality. Identifies where and why robot perception diverged from expectations.
Causal reasoning engine, multi-factor causality analysis, incident narratives, evidence quality scoring, and LLM-assisted root cause analysis with semantic search.
OKF-inspired temporal knowledge system. Entities persist across missions. Track location history, temporal facts, anomaly records. Enable: "Pallet moved from aisle_3 to aisle_5."
Ground entities in X,Y,Z coordinates with movement vectors and trends. Track "moved 2.3m northeast" not just "moved."
Longitudinal reasoning across mission sequences. Predict entity behavior, detect environmental evolution, enable cross-mission pattern detection.
Swappable detection backends (YOLO for speed, SAM for zero-shot flexibility, template fallback for offline). Automatic fallback chain ensures always-working detection.
Terrain-aware perception. Track zone traversability, assess entity risk by terrain type. Multi-robot fleet consensus on zone difficulty. Anomaly detection at fleet scale.
Open-vocabulary object detection for invisible object discovery. Segment anything model for precise boundaries. Compare YOLO vs DINO to identify perception gaps. Context-aware severity scoring with terrain and historical data.
RGB + thermal/infrared fusion for offline forensic reconstruction. Identify invisible persons in low-light, smoke, fog, shadows, occlusions. Root cause analysis, sensor disagreement detection, recommendations for future systems.
Multi-modal data ingestion for heterogeneous sources: ROS 2 bags, video, Linux system logs, Nav2 exports, point clouds, operator annotations, sensor calibration. Unified timeline with automatic clock synchronization. Handles time model detection (ROS nanoseconds, wall-clock, frame numbers, sequences) and temporal alignment across all modalities.
AI-powered navigation failure analysis across 7 dimensions: localization (AMCL divergence, odometry drift), planner (oscillation, deadlock), costmap (inflation, conflicts), dynamic obstacles, semantic gaps, environmental context, controller stability. Distinguishes Nav2 architectural limitations from tuning/environment issues. Generates structured findings with tiered recommendations (tuning/capability/architecture) and confidence scoring (0.0-1.0) based on evidence strength.
pip install pyroboreplay==2.1.0
# or with uv
uv install pyroboreplay==2.1.0
# From source
git clone https://git.example.com/user/pyroboreplay.git
cd pyroboreplay
cargo build --release# Interactive timeline scrubber
pyroboreplay replay mission.bag
# Complete forensic investigation
pyroboreplay forensic mission.bag --output investigation.json
# Sensor fusion analysis (RGB + thermal)
pyroboreplay fuse-sensors rgb.bag thermal.bag --report forensic.md
# Cross-mission pattern detection
pyroboreplay cross-mission *.bag --learn-patterns --predict-nextKeyboard shortcuts:
- Space: Play/Pause | /: Step | Ctrl+J: Jump | f: Filter | q: Quit
from pyroboreplay import Mission, ForensicAnalyzer, RGBThermalFusion
# Load mission
mission = Mission.from_ros_bag("warehouse.bag")
# Causal analysis
hypothesis = mission.analyze_failure(timestamp=1234567890)
print(f"Root cause: {hypothesis.description}")
print(f"Confidence: {hypothesis.confidence:.0%}")
# Invisible object detection (DINO + SAM)
retrospective = mission.analyze_retrospectively()
print(f"Objects RGB missed: {len(retrospective.gaps)}")
for gap in retrospective.gaps:
print(f" {gap.dino_detection.class_name} at risk level {gap.severity}")
# Multispectral forensic analysis
fusion = RGBThermalFusion()
fusion.load_rgb_detections(mission.rgb_detections)
fusion.load_thermal_frame(thermal_data)
fusion.fuse()
stats = fusion.get_statistics()
print(f"Thermal-only detections: {stats.thermal_only_detections}")
print(f"RGB miss rate: {stats.rgb_miss_rate:.1%}")
print(f"Confidence improvement: +{stats.confidence_improvement:.1%}")
# Persistent world knowledge (cross-mission learning)
world_model = mission.extract_world_knowledge()
print(f"Known entities: {len(world_model.entities)}")
print(f"Temporal facts recorded: {len(world_model.temporal_facts)}")
# Next mission prediction
prediction = mission.predict_next_mission()
print(f"Likely obstacles: {prediction.expected_obstacles}")| Feature | v0.1 | v0.5 | v0.9 | v1.0 | v2.0 | v2.1 |
|---|---|---|---|---|---|---|
| Sensor Replay | A | A | A | A | A | A |
| Timeline Queries | - | A | A | A | A | A |
| Causal Analysis | - | - | A | A | A | A |
| Root Cause Diagnosis | - | - | A | A | A | A |
| Cross-Mission Learning | - | - | - | A | A | A |
| Pluggable Detection (YOLO/SAM) | - | - | - | - | A | A |
| Terrain Intelligence | - | - | - | - | A | A |
| Persistent World Knowledge | - | - | - | - | A | A |
| Retrospective Object Discovery | - | - | - | - | A | A |
| Multispectral Sensor Fusion | - | - | - | - | A | A |
| Forensic Investigation Reports | - | - | - | - | A | A |
| Fleet Learning & Consensus | - | - | - | - | A | A |
| Invisible Person Detection | - | - | - | - | A | A |
| Universal Temporal Fusion | - | - | - | - | - | A |
| Multi-Modal Data Ingestion | - | - | - | - | - | A |
| Root Cause Inference Engine | - | - | - | - | - | A |
| Nav2 Limitation Detection | - | - | - | - | - | A |
| Semantic Gap Analysis | - | - | - | - | - | A |
| 647 Comprehensive Tests | - | - | - | - | - | A |
Debug fleet behavior, identify missed detections, optimize coverage.
# Find what robot missed (RGB vs thermal)
pyroboreplay invisible-persons warehouse.bag --thermal-data warehouse.thermalResult: Identify undetected workers, near-collisions, coverage gaps.
Verify inspection coverage, detect missed areas, analyze sensor performance.
# Multispectral analysis with RGB + thermal
pyroboreplay fuse-sensors rgb_survey.bag thermal_survey.bagResult: Find unscanned areas invisible to standard RGB.
Compare perception strategies, analyze fleet behavior, identify sim-to-reality gaps.
exp_a = Mission.from_bag("strategy_v1.bag")
exp_b = Mission.from_bag("strategy_v2.bag")
# Causal analysis
causes_a = exp_a.root_cause_analysis()
causes_b = exp_b.root_cause_analysis()
improvement = len(causes_a) - len(causes_b)
print(f"v2 fixes {improvement} issues vs v1")Result: Data-driven strategy selection, quantified improvements.
Verify robot didn't miss people, generate forensic reports, audit sensor performance.
# Complete forensic investigation
pyroboreplay forensic operation.bag --output compliance_report.jsonResult: Provable safety assurance, auditable incident investigation.
Mission Data Input (ROS 2 Bag / Gazebo / Simulation)
|
v
Phases 1-4: Reality Gap Detection
|-- Probabilistic gap scoring
|-- Severity classification
|-- Historical findings database
|-- Evidence aggregation
|
v
Phases 5-9: Intelligent Analysis
|-- Causal event graphs
|-- Multi-factor causality
|-- Incident narratives
|-- Evidence quality scoring
|-- LLM-assisted root cause analysis
|-- Semantic search
|
v
Phases 10-11: Temporal Knowledge + Terrain Intelligence
|-- Persistent world model (entities, locations, facts)
|-- Spatial grounding (x,y,z coordinates)
|-- Multi-mission learning (longitudinal reasoning)
|-- Terrain zones and traversability
|-- Fleet learning (multi-robot consensus)
|
v
Phase 7 Enhanced: Pluggable Detection
|-- YOLO backend (real-time)
|-- SAM backend (zero-shot)
|-- Template fallback (offline)
|-- Orchestrator (automatic fallback)
|
v
Phase 12: Retrospective DINO + SAM
|-- Open-vocabulary object detection
|-- Segment anything model
|-- Invisible object discovery
|-- Context-aware gap analysis
|-- Recommendations engine
|
v
Phase 13: Multispectral Sensor Fusion
|-- Thermal imaging model
|-- RGB-Thermal fusion engine
|-- Invisible person detection (17 scenarios)
|-- Forensic report generation
|-- Root cause analysis
|
v
Output: Forensic Reports, Recommendations, Predictions
Key Innovation: Each phase builds on prior layers. Real-time detection (Phase 7) feeds offline analysis (Phases 12-13). Offline findings improve world knowledge (Phase 10). World knowledge informs next mission (Phases 10.3, 6).
| Metric | Target | v2.0 Status |
|---|---|---|
| Mission ingestion | 10k events/sec | Tested |
| Timeline scrubbing | <100ms latency | Optimized |
| Large mission queries (1M events) | <1s | Passing |
| Forensic analysis (full pipeline) | <5s | Achieved |
| Multispectral fusion | <2s per frame | Efficient |
Test Coverage: 558 passing tests
- Phases 1-4: 60 tests
- Phases 5-9: 140 tests
- Phase 10 (Knowledge): 26 tests
- Phase 7 Enhanced: 15 tests
- Phase 11: 17 tests
- Phase 12: 20 tests
- Phase 13: 18 tests
- Integration & edge cases: 246 tests
cargo build --release
maturin develop # Install Python wheel# Full test suite
cargo test
# By phase
cargo test phase_13 # Multispectral fusion
cargo test phase_12 # Retrospective detection
cargo test phase_11 # Terrain intelligence
cargo test knowledge # Persistent world model
# Examples
cargo run --example forensic_analysis_demo
cargo run --example thermal_fusion_democargo clippy --all-targets -- -D warnings
cargo fmt --check
cargo audit- CLAUDE.md — Complete product vision & architecture
- Examples — Working demos (replay, forensics, fusion, cross-mission learning)
- API Reference — Python & Rust APIs
- Phases Overview — Detailed phase descriptions
We welcome contributions! See CONTRIBUTING.md for development setup, coding conventions, and PR guidelines.
Easiest ways to help:
- Report bugs or feature ideas: GitHub Issues
- Share how you're using PyRoboReplay: GitHub Discussions
- Star the repo if it helps you
Open Source Software — Use freely in academic, commercial, and personal projects. See LICENSE.
If PyRoboReplay helps your research or product, please star the repo and cite:
@software{pyroboreplay2026,
title={PyRoboReplay: Forensic Debugging and Multispectral Analysis for Autonomous Robots},
author={Mullassery, Georgi},
year={2026},
version={2.0.0},
url={https://git.example.com/user/pyroboreplay}
}New to robot debugging? Start with the quick start above.
Ready for production? Check out the architecture and examples.
Have questions? Open an issue or discussion.
Built for robotics teams who demand understanding, not just visibility.
PyRoboReplay: Because great robots are built on knowledge, not intuition.
If this helps you, please star the repo!