Yimo/feat/host fragment - #3967
Conversation
Greptile SummaryThis change adds distributed Host lifecycle control, deployment supervision, and hosted fragment compilation. Two reproduced failures need correction before merging:
T-Rex validation blockedThe focused class-reference compatibility check could not execute because the environment is missing the Confidence Score: 1/5Not safe to merge: the lifecycle control endpoint permits unsafe deserialization, and a stale startup failure can stop a replacement deployment. The review contains three blocking findings, including one security issue. Two were reproduced through focused execution; the remaining compiler compatibility issue is supported by the differing resolver behavior but could not complete its focused execution check. Files Needing Attention: dimos/cli/commands/host.py, dimos/hosted/daemon.py, and dimos/hosted/fragment_compiler.py
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| cleanup.callback(rpc.stop) | ||
| control_name = HOST_CONTROL_RPC_NAME.format(host_id=host_id) | ||
| rpc.serve_rpc(daemon.describe, f"{control_name}/describe") # type: ignore[arg-type] | ||
| rpc.serve_rpc(daemon.start, f"{control_name}/start") # type: ignore[arg-type] |
There was a problem hiding this comment.
If an untrusted peer can reach the shared Zenoh fabric, it can send a request to this Host start endpoint. The request is pickle-deserialized before HostDaemon.start checks its epoch or fragment metadata, and there is no caller authentication or authorization boundary. A peer can therefore trigger arbitrary pickle reconstruction and execute code in the Host service.
How this was verified: A payload reducer executed before the lifecycle handler returned its stale-epoch validation error.
Knowledge Base Used:
Artifacts
- Focused validation source that sends the lifecycle request with a controlled pickle reducer, demonstrating that payload reconstruction occurs before lifecycle validation.
Direct lifecycle validation output
- Direct stale-epoch invocation output establishes that the handler validation would reject the request after it is invoked.
- Anonymous lifecycle request output shows the pickle reducer marker appears before the stale-epoch handler error, confirming unsafe deserialization occurs first.
| error = self._wait_for_start(receive_ready) | ||
| if error is not None: | ||
| _terminate(process, self._stop_timeout) | ||
| kill_run_processes(fragment.run_id) |
There was a problem hiding this comment.
A deployment can be stopped and replaced while its original start() call is still waiting for readiness. If that stale call then times out or receives EOF, this run-wide cleanup kills every process with the same DIMOS_RUN_ID, including the replacement deployment. The identity check on line 167 happens only after the sweep, so it cannot prevent the failure. Check that the deployment is still current before sweeping, or limit cleanup to the failed deployment's process tree.
Knowledge Base Used:
Artifacts
Stale startup reproduction source
- Focused concurrency reproduction source that creates a stale startup and a replacement deployment sharing one run ID.
Stale startup reproduction command
- Command wrapper used to execute the stale-start replacement scenario.
Replacement termination output
- Execution output records the stale cleanup sweep terminating the replacement PID and the replacement ending in a failed state.
| if not possible: | ||
| disabled = any( | ||
| atom.module in blueprint.disabled_modules_tuple and _matches_reference(atom, requested) | ||
| for atom in blueprint.blueprints | ||
| ) | ||
| if reference.optional or disabled: | ||
| return None | ||
| raise ValueError( | ||
| f"Module reference {consumer.name}.{reference.name} has no provider in the Blueprint" | ||
| ) |
There was a problem hiding this comment.
When a required class-typed module reference has no provider in the blueprint, the fragment compiler raises an error. The existing coordinator instead preserves the class-reference fallback and accepts the same local blueprint. Hosted compilation can therefore reject existing graphs that previously built successfully. Keep this resolver aligned with _resolve_single_ref, or explicitly validate and document the compatibility break.
Knowledge Base Used:
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
1 participant
This PR adds the core infrastructure for deploying a DimOS Blueprint across multiple Hosts:
.hosted()to describe where a module should run;HostFragmentobjects;.hosted()API.hosted()can be applied to a single module or a composed Blueprint. A composed Blueprint becomes one placement unit and is assigned to a single Host.Core Behavior
dimos/runs/<run-id>/streams/<stream>/<message-type>;run_id,generation,host_id, application revision, and SHA-256 digest;This PR does not include code distribution, failover, migration, replicas, or resource-aware scheduling. The standard
dimos runcommand is also not yet connected to the full discovery → placement → dispatch workflow.Validation on
yimo/temp/host_e2eThis branch adds a Host client, scenario runner, and Docker Compose environment on top of the feature PR.
1. Switch Branches
2. Run the Three Automated Scenarios
basicedge-a; transform →compute-a; sink → controllerreplayreplay-a; transform →compute-a; sink → controllersimsim-a; odom probe →compute-a; sink → controllerCompose topology
flowchart LR C["verify-* / Controller"] -->|"discover + lifecycle RPC"| Z["Zenoh Router"] Z <--> E["edge-host\nedge-a / edge"] Z <--> G["compute-host\ncompute-a / compute,gpu"] Z <--> R["replay-host\nreplay-a / replay"] Z <--> S["sim-host\nsim-a / sim"] E <-->|"run-scoped streams"| G G <-->|"run-scoped streams"| C R <-->|"run-scoped streams"| G S <-->|"run-scoped streams"| GThe Compose file has three layers:
edge-host,compute-host,replay-host, andsim-hosteach run an independentHostDaemonwith a stable name and tags;verify-*container owns the original Blueprint, discovers Hosts, resolves placement, compiles Fragments, and checks the final application result.All services use the same locally built image, so every Host has the same code and application revision. They share the same runtime configuration: Zenoh transport, client mode, explicit connection to
tcp/router:7447, two local workers, and no multicast discovery.sim-hostand the simulation controllers also receive additional shared memory for MuJoCo.Control plane and data plane
The controller uses the router to discover Host liveliness tokens and call Host RPCs such as
describe,start, andstop. This is the control plane.After deployment, application messages do not pass through the controller as a proxy. The Fragment compiler pins every boundary stream to a run-scoped Zenoh key, and the modules exchange data directly through the router. This is the data plane.
Starting consumers first prevents short-lived replay or source messages from being published before their remote subscribers are ready.
Scenario module graphs
Each profile changes only the scenario controller; the router and four Host services keep the same topology.
basicexercises exact-name placement, tag placement, the local default, and two Host boundaries.replayreplaces the synthetic source with a real SQLite-backed replay.simreplaces it with a real headless MuJoCo process and a typedPoseStampedodometry stream.The
ValidationSinkis deliberately local and is the end-to-end assertion point. It writes a result only after all required values arrive. The controller then prints the resolved placement and application result, stops each Fragment using its run/generation/digest identity, and exits with the scenario status. The shell runner propagates that status through--exit-code-fromand removes the Compose containers and network.The scenarios can also be run individually:
3. Run the Visual Scenario
After
VISUAL:READYappears, open:The visual profile extends the simulation graph with a controller-local Rerun bridge:
MuJoCo updates the robot's real root pose and velocity on
sim-a. The same run-scoped odometry stream is consumed by the probe for an automated assertion and by Rerun for visualization. Static ground and target geometry are logged by the bridge, while the robot body is attached to the live odometry transform.Unlike the finite profiles,
visualdoes not exit after validation.VISUAL:READYmeans Host discovery, placement, Fragment startup, cross-Host odometry, and the application-level assertion have already succeeded; the controller is kept alive so the browser can remain connected.The blue robot should move continuously in a circle, with the yellow arrow rotating to show its direction of travel. The red target and ground remain fixed, making motion easy to verify.