Skip to content

feat(microduck): browser cockpit on the official-policy sim - #4099

Draft
spomichter wants to merge 8 commits into
mainfrom
feat/microduck-cockpit
Draft

spomichter wants to merge 8 commits into
mainfrom
feat/microduck-cockpit

Conversation

@spomichter

Copy link
Copy Markdown
Contributor

Stacked on #3937 (feat/microduck-sim, merged with main); the diff shrinks to the two top commits once that lands.

What

  • feat(web): Map2D click-to-goalMap2D(goal="goal_request") binds a third channel (tx, goal.json.v1, publish="shared") through the existing panel/publish path. A map click publishes {x, y}; the bridge decodes it to a PoseStamped on a generated Out port that autoconnects to the planner's goal_request. View-only maps are unchanged. No new protocol messages, panel kinds or relay changes.
  • feat(microduck): browser cockpit blueprintsmicroduck-cockpit = Feat/microduck sim #3937's sim + policy task, plus voxel mapper, costmapper, A* planner, movement manager, and a three-panel cockpit (camera, click-to-goal map, keyboard teleop). microduck-agentic-cockpit adds McpClient, observe, chat, and a small MicroduckSkills module (go_to, stop_moving, where_am_i, list_tricks, perform, sit, stand_up) over the policy task's RPCs. simulation.py gains microduck_stack(sim_overrides) so the cockpit can enable the pointcloud without duplicating the wiring.

Run

uv run dimos --simulation mujoco run microduck-cockpit --local-relay
uv run dimos --simulation mujoco run microduck-agentic-cockpit --local-relay   # needs OPENAI_API_KEY

Verified

  • Headless smoke run: sim, coordinator, 7 policies armed, mapper, planner, relay up in ~5 s. A Python viewer published a goal through the relay, got pub_ack, planner walked the duck to (0.6, 0.2).
  • pytest: dimos/web, dimos/robot/pollen/microduck, policy task, adapters, blueprint checks (767 passed). mypy clean on changed files. Registry regenerated.
  • Deno shared tests, cockpit + SDK vitest, tsc clean. web/ formatted with the pinned Deno 2.6.10.

Nabla7 and others added 8 commits September 4, 2026 09:46
Map2D(goal=...) binds a third, tx channel (goal.json.v1, publish=shared)
that a map click publishes {x, y} on; the bridge decodes it to a
PoseStamped on a generated Out port that autoconnects to the planner's
goal_request. View-only maps are unchanged.
microduck-cockpit: the official-policy sim (microduck-sim) plus the
mapping/planning stack and a three-panel cockpit (camera, click-to-goal
map, keyboard teleop). microduck-agentic-cockpit adds McpClient, the
observe skill, a small MicroduckSkills module (go_to, stop_moving,
where_am_i, list_tricks, perform, sit, stand_up over the policy task's
RPCs) and the chat panel.

simulation.py gains microduck_stack(sim_overrides) so a cockpit can turn
the pointcloud on without duplicating the sim/coordinator wiring.
@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
5468 3 5465 51
View the top 3 failed test(s) by shortest run time
dimos.codebase_checks.test_docs_branding::test_docs_use_current_branding
Stack Traces | 0.038s run time
def test_docs_use_current_branding() -> None:
        """Fail if any file under docs/ spells the brand "DimOS" instead of "dimOS"."""
        hits = find_old_branding()
        if hits:
            listing = "\n".join(
                f"  - {p.relative_to(DIMOS_PROJECT_ROOT)}:{lineno}: {line.strip()}"
                for p, lineno, line in hits
            )
>           raise AssertionError(f'Found "DimOS" in docs/:\n{listing}\n\nThe brand is spelled "dimOS".')
E           AssertionError: Found "DimOS" in docs/:
E             - docs/usage/microduck-simulation.md:59: MuJoCo is always headless in this blueprint. `--viewer` controls DimOS
E             - docs/usage/microduck-simulation.md:83: Focus the DimOS Rerun viewer and use W/S for forward/reverse, Q/E for strafe,
E           
E           The brand is spelled "dimOS".

hits       = [(PosixPath('.../docs/usage/microduck-simulation.md'), 59, 'MuJoCo is always headless in thi.../usage/microduck-simulation.md'), 83, 'Focus the DimOS Rerun viewer and use W/S for forward/reverse, Q/E for strafe,')]
listing    = '  - docs/usage/microduck-simulation.md:59: MuJoCo is always headless in this blueprint. `--viewer` controls DimOS\n  - docs/usage/microduck-simulation.md:83: Focus the DimOS Rerun viewer and use W/S for forward/reverse, Q/E for strafe,'

dimos/codebase_checks/test_docs_branding.py:43: AssertionError
dimos.hardware.test_adapter_registries::test_declared_names_match_golden_set[whole_body]
Stack Traces | 0.09s run time
registry = <dimos.hardware.whole_body.registry.WholeBodyAdapterRegistry object at 0x7fd6392e9090>
family = 'whole_body'

    @pytest.mark.parametrize(("registry", "family"), FAMILIES)
    def test_declared_names_match_golden_set(registry, family) -> None:
>       assert set(registry._factory_paths) == EXPECTED_NAMES[family]
E       AssertionError: assert {'dual_openya...croduck', ...} == {'dual_openya...ort_lcm', ...}
E         
E         Extra items in the left set:
E         #x1B[0m#x1B[33m'#x1B[39;49;00m#x1B[33msim_mujoco_microduck#x1B[39;49;00m#x1B[33m'#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         
E         Full diff:
E         #x1B[0m#x1B[90m #x1B[39;49;00m {#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     'dual_openyam_damiao',#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     'mock_whole_body',#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     'openarm_damiao',#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     'openyam_damiao',#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     'sim_mujoco_g1',#x1B[90m#x1B[39;49;00m
E         #x1B[92m+     'sim_mujoco_microduck',#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     'transport_lcm',#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     'transport_ros',#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m }#x1B[90m#x1B[39;49;00m

family     = 'whole_body'
registry   = <dimos.hardware.whole_body.registry.WholeBodyAdapterRegistry object at 0x7fd6392e9090>

dimos/hardware/test_adapter_registries.py:106: AssertionError
dimos.robot.pollen.microduck.test_simulation::test_scene_package_uses_standard_mujoco_composition
Stack Traces | 3.3s run time
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fe22f5cc310>
tmp_path = PosixPath('.../pytest-0/popen-gw3/test_scene_package_uses_standa0')

    def test_scene_package_uses_standard_mujoco_composition(
        monkeypatch: pytest.MonkeyPatch,
        tmp_path: Path,
    ) -> None:
        scene_xml = tmp_path / "scene.xml"
        package = ScenePackage(
            package_dir=tmp_path,
            source_path=tmp_path / "source.glb",
            alignment=SceneMeshAlignment(),
            mujoco_scene_path=scene_xml,
            entities=[{"id": "test_entity"}],
        )
        monkeypatch.setattr(
            microduck_blueprint,
            "resolve_scene_package",
            lambda _scene: package,
        )
    
        backend, adapter_address = microduck_blueprint._microduck_mujoco_backend("office")
        simulator = backend.active_blueprints[0]
    
        assert simulator.module is MujocoSimModule
        assert simulator.kwargs["scene_xml"] == scene_xml
>       assert simulator.kwargs["robot_mjcf"] == MICRODUCK_ROBOT_MJCF

adapter_address = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/microduck.tar.gz after 3 attempts: Com...icroduck.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] LfsPath object at 0x7fe3f61c8350>
backend    = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/microduck.tar.gz after 3 attempts: Com...roduck.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] Blueprint object at 0x7fe22f5cf790>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fe22f5cc310>
package    = ScenePackage(package_dir=PosixPath('.../pytest-0/popen-gw3/test_scene_package_use..., mujoco_binary_path=None, mujoco_composed_binaries={}, metadata_path=None, entities=[{'id': 'test_entity'}], stats={})
scene_xml  = PosixPath('.../pytest-0/popen-gw3/test_scene_package_uses_standa0/scene.xml')
simulator  = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/microduck.tar.gz after 3 attempts: Com...ck.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] BlueprintAtom object at 0x7fe22f5cec50>
tmp_path   = PosixPath('.../pytest-0/popen-gw3/test_scene_package_uses_standa0')

.../pollen/microduck/test_simulation.py:114: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../...../_temp/uv-python-dir/cpython-3.10-linux-x86_64-gnu/lib/python3.10/pathlib.py:666: in __eq__
    return self._cparts == other._cparts and self._flavour is other._flavour
        other      = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/microduck.tar.gz after 3 attempts: Com...icroduck.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] LfsPath object at 0x7fe3f61c8350>
        self       = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/microduck.tar.gz after 3 attempts: Com...icroduck.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] LfsPath object at 0x7fe3f61c8350>
dimos/utils/data.py:370: in __getattribute__
    resolved = object.__getattribute__(self, "_ensure_downloaded")()
        name       = '_cparts'
        self       = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/microduck.tar.gz after 3 attempts: Com...icroduck.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] LfsPath object at 0x7fe3f61c8350>
dimos/utils/data.py:353: in _ensure_downloaded
    cache = get_data(filename)
        cache      = None
        filename   = 'microduck/robot_groundcontact.xml'
        self       = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/microduck.tar.gz after 3 attempts: Com...icroduck.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] LfsPath object at 0x7fe3f61c8350>
dimos/utils/data.py:310: in get_data
    archive_path = _decompress_archive(_pull_lfs_archive(archive_name))
        archive_name = 'microduck'
        data_dir   = PosixPath('.../dimos/dimos/data')
        file_path  = PosixPath('.../dimos/dimos/data/microduck/robot_groundcontact.xml')
        name       = 'microduck/robot_groundcontact.xml'
        nested_path = PosixPath('robot_groundcontact.xml')
        path_parts = ('microduck', 'robot_groundcontact.xml')
dimos/utils/data.py:254: in _pull_lfs_archive
    _lfs_pull(file_path, repo_root)
        file_path  = PosixPath('.../dimos/data/.lfs/microduck.tar.gz')
        filename   = 'microduck'
        repo_root  = PosixPath('.../work/dimos/dimos')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

file_path = PosixPath('.../dimos/data/.lfs/microduck.tar.gz')
repo_root = PosixPath('.../work/dimos/dimos')

    def _lfs_pull(file_path: Path, repo_root: Path, *, retries: int = 2) -> None:
        relative_path = file_path.relative_to(repo_root)
    
        env = os.environ.copy()
        env["GIT_LFS_FORCE_PROGRESS"] = "1"
    
        last_err: subprocess.CalledProcessError | None = None
        for attempt in range(1, retries + 2):  # retries + 1 total attempts
            try:
                subprocess.run(
                    # --exclude= overrides lfs.fetchexclude from .lfsconfig, which
                    # otherwise silently skips data/.lfs/* even when --include matches.
                    ["git", "lfs", "pull", "--include", str(relative_path), "--exclude="],
                    cwd=repo_root,
                    check=True,
                    env=env,
                )
                return
            except subprocess.CalledProcessError as e:
                last_err = e
                if attempt <= retries:
                    time.sleep(attempt)  # 1s, 2s backoff
    
>       raise RuntimeError(
            f"Failed to pull LFS file {file_path} after {retries + 1} attempts: {last_err}"
        )
E       RuntimeError: Failed to pull LFS file .../dimos/data/.lfs/microduck.tar.gz after 3 attempts: Command '['git', 'lfs', 'pull', '--include', 'data/.lfs/microduck.tar.gz', '--exclude=']' returned non-zero exit status 1.

attempt    = 3
env        = {'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LT...-version=2.0', 'ACTIONS_ORCHESTRATION_ID': 'a6701975-88da-449b-b4ec-b819e5769beb.tests._3_10_ubuntu-latest_false', ...}
file_path  = PosixPath('.../dimos/data/.lfs/microduck.tar.gz')
last_err   = CalledProcessError(1, ['git', 'lfs', 'pull', '--include', 'data/.lfs/microduck.tar.gz', '--exclude='])
relative_path = PosixPath('data/.lfs/microduck.tar.gz')
repo_root  = PosixPath('.../work/dimos/dimos')
retries    = 2

dimos/utils/data.py:224: RuntimeError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants