diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 25c98d1988..0a63121303 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -495,6 +495,9 @@ jobs: dimos/e2e_tests/test_robot_picker_browser.py dimos/e2e_tests/test_relay_auth_browser.py --no-cov + - name: Relay image build # docker/relay/Dockerfile must not rot silently + run: docker build -f docker/relay/Dockerfile . + tests: if: | !cancelled() && diff --git a/.gitignore b/.gitignore index c3be95f27f..2963cefeec 100644 --- a/.gitignore +++ b/.gitignore @@ -28,6 +28,9 @@ build/ /web/cockpit/.dist-* /web/sdk/.dist-* +# Hosted relay: certificate and auth file next to docker/relay/compose.yaml +/docker/relay/config/ + # Nix / direnv (symlink one of the tracked .envrc.* files to .envrc) .direnv/ .envrc diff --git a/docker/relay/Dockerfile b/docker/relay/Dockerfile new file mode 100644 index 0000000000..99f2175080 --- /dev/null +++ b/docker/relay/Dockerfile @@ -0,0 +1,27 @@ +# The hosted relay: web/ built into one image, run as the image's unprivileged +# deno user (uid 1993) with the certificate and auth file mounted at /etc/relay. +# Guide: docs/usage/relay_hosting.md; compose.yaml next to this file runs it. +# The Deno pin lives in dimos/utils/deno.py (CI greps it there); keep them equal. +FROM denoland/deno:2.6.10 + +# A server has no use for the daily version check against dl.deno.land. +ENV DENO_NO_UPDATE_CHECK=1 +WORKDIR /app/web +# The build context is the repo root; Dockerfile.dockerignore next to this +# file narrows it to web/ (the root .dockerignore would drop package.json). +# WORKDIR created /app/web as root and --chown covers only the copied entries, +# so the directory itself needs the chown for deno to create node_modules. +COPY --chown=deno:deno web/ /app/web +RUN chown deno:deno /app/web +USER deno +RUN deno install --frozen && deno task -r build + +EXPOSE 443/tcp 443/udp +# Docker lets the unprivileged user bind 443 (ip_unprivileged_port_start=0 +# inside containers since 20.10). CMD holds only the file flags, so a compose +# `command:` can override those without repeating the rest. +ENTRYPOINT ["deno", "run", "--frozen", "--allow-net", "--allow-read=/app/web,/etc/relay", \ + "relay/main.ts", "--host", "0.0.0.0", "--port", "443", \ + "--cockpit-dir", "cockpit/dist", "--sdk-dir", "sdk/dist"] +CMD ["--cert", "/etc/relay/fullchain.pem", "--key", "/etc/relay/privkey.pem", \ + "--auth-file", "/etc/relay/auth.json"] diff --git a/docker/relay/Dockerfile.dockerignore b/docker/relay/Dockerfile.dockerignore new file mode 100644 index 0000000000..4dc23b33e8 --- /dev/null +++ b/docker/relay/Dockerfile.dockerignore @@ -0,0 +1,15 @@ +# Replaces the repo-root .dockerignore for docker/relay/Dockerfile (BuildKit +# prefers .dockerignore): that one drops package.json, which the +# web build needs. Only web/ is copied; skip build products and local secrets. +* +!web +web/**/node_modules +web/**/dist +web/**/.dist-* +web/**/.build.lock +web/**/.env +web/**/.env.* +web/**/auth.json +web/**/*.pem +web/**/*.key +web/**/*.crt diff --git a/docker/relay/compose.yaml b/docker/relay/compose.yaml new file mode 100644 index 0000000000..df3a5a4786 --- /dev/null +++ b/docker/relay/compose.yaml @@ -0,0 +1,18 @@ +# The hosted relay (docs/usage/relay_hosting.md). From a checkout: +# docker compose -f docker/relay/compose.yaml up -d --build +# config/ (gitignored) holds fullchain.pem, privkey.pem, and auth.json, owned +# by uid 1993 (the image's deno user). The relay reads them at startup only. +services: + relay: + image: dimos-relay + build: + context: ../.. + dockerfile: docker/relay/Dockerfile + ports: + - "443:443/tcp" + - "443:443/udp" + volumes: + - ./config:/etc/relay:ro + restart: unless-stopped + # The relay closes its sessions on SIGTERM and exits at once. + stop_grace_period: 5s diff --git a/docs/usage/relay_hosting.md b/docs/usage/relay_hosting.md new file mode 100644 index 0000000000..a3e1de13f1 --- /dev/null +++ b/docs/usage/relay_hosting.md @@ -0,0 +1,162 @@ +# Relay hosting + +The relay on your own server, under your own name: robots dial out to it with a key, operators open it in a browser with a token, and the cockpit works from anywhere. This guide takes a fresh Ubuntu VM and a domain to `https://dimos-relay.example.com`. Section 9 does the same on a LAN without a domain. The relay itself needs nothing new; everything here is packaging: a container image ([`docker/relay/Dockerfile`](/docker/relay/Dockerfile)), a compose file ([`docker/relay/compose.yaml`](/docker/relay/compose.yaml)), and the steps around them. The flags it uses are explained in [Web SDK](/docs/usage/web_sdk.md). + +## 1. What you get, and what you do not + +- One relay process on one machine. The registry is in memory: after a restart every robot re-registers by itself within seconds and every open page reconnects. +- No accounts. The auth file holds robot keys, each bound to one robot id, and viewer tokens. A viewer token does everything (view, teleop, publish) on every robot behind the relay, so hand tokens out like passwords. +- The relay terminates TLS itself, on one port for HTTPS (TCP) and QUIC (UDP). No reverse proxy: an HTTP proxy cannot carry QUIC, and the relay advertises the WebTransport URL from the host the client dialed. +- The certificate and the auth file are read at startup. Changing either means a restart, which costs a few seconds of reconnection. + +## 2. DNS and firewall + +Point an A record at the VM (`dimos-relay.example.com -> `). Open 443 TCP, 443 UDP, and 80 TCP for certbot (nothing listens on 80 between renewals). The cloud provider's own firewall (security group) needs the same rules; UDP 443 is the one people forget. + +```bash +sudo ufw allow 22/tcp && sudo ufw allow 80/tcp && sudo ufw allow 443/tcp && sudo ufw allow 443/udp +sudo ufw enable +``` + +Without UDP 443 there is no WebTransport and, by design, no fallback: the page loads and never connects. + +## 3. Certificate + +Clone dimos onto the VM (the image is built from the checkout) and install Docker and certbot. Request the certificate with a deploy hook that installs the two PEM files into the relay's config directory, owned by uid 1993 (the image's `deno` user), and restarts the relay: + +```bash +sudo apt install -y git certbot docker.io docker-compose-v2 +sudo git clone https://github.com/dimensionalOS/dimos /srv/dimos +sudo mkdir /srv/dimos/docker/relay/config +sudo certbot certonly --standalone -d dimos-relay.example.com --deploy-hook \ + 'install -o 1993 -g 1993 -m 600 "$RENEWED_LINEAGE"/fullchain.pem "$RENEWED_LINEAGE"/privkey.pem /srv/dimos/docker/relay/config/ && docker compose -f /srv/dimos/docker/relay/compose.yaml restart relay' +``` + +The hook runs now (so `config/` holds `fullchain.pem` and `privkey.pem`; the restart is a no-op until the relay exists) and certbot saves it in the renewal configuration: every automatic renewal (certbot's timer, roughly every 60 days) installs the new files and restarts the relay, which reads certificates only at startup. `sudo certbot renew --dry-run` checks renewal but skips hooks, so rehearse the restart once by running the hook command by hand after step 5. + +## 4. Auth file + +`config/auth.json` maps robot ids to keys and viewer names to tokens, each 16 to 256 characters and no secret used twice. Generate them with `openssl rand -hex 32`: + +```json +{ + "robots": { "go2-lab": "" }, + "viewers": { "paul": "" } +} +``` + +```bash +sudo chown 1993:1993 /srv/dimos/docker/relay/config/auth.json +sudo chmod 600 /srv/dimos/docker/relay/config/auth.json +``` + +A robot id here has to match the robot's `--robot-id` (its hostname when not given). Adding or rotating a secret is an edit plus `docker compose restart relay`. The relay refuses a malformed file with a message that names the entry, never the secret, and it never logs secrets. + +## 5. Run + +```bash +cd /srv/dimos/docker/relay +sudo docker compose up -d --build +curl https://dimos-relay.example.com/api/info +sudo docker compose logs -f relay +``` + +The first build takes a few minutes: Deno installs the web dependencies and builds the SDK and the cockpit inside the image. `/api/info` answers `{"wtUrl":"https://dimos-relay.example.com","v":...}`. The log opens with the ready line and then shows connections (`[relay] robot connected`, `[relay] viewer 1 authenticated as paul`), rejections with their reason (`[relay] robot go2-lab rejected: invalid robot key`), and `[relay] robot go2-lab disconnected` when one leaves. `restart: unless-stopped` brings the relay back after a reboot. A new dimos version is `git pull` and the same `up -d --build`. + +## 6. Robot + +The robot needs the key bound to its id, from the environment or the `.env` file of the checkout it runs from (the `--relay-key` flag exists but shows in the process list): + +```bash +RELAY_KEY= dimos run unitree-go2 --relay-url https://dimos-relay.example.com --robot-id go2-lab +``` + +It dials out, so it works behind NAT with no open ports. A wrong key is logged once as `auth_failed` and the bridge gives up rather than retrying: fix `RELAY_KEY` and start again. + +## 7. Operator + +Open `https://dimos-relay.example.com/`. The page asks for a viewer token, keeps it in the browser's local storage, and "Log out" in the status bar forgets it. With several robots registered the page lists them; "switch robot" reopens the list. `/api/stats` (per-robot, per-channel, and per-viewer counters, with viewer names) takes the same token as a bearer: + +```bash +curl -H "Authorization: Bearer " https://dimos-relay.example.com/api/stats +``` + +## 8. Without Docker + +The same relay under systemd: the pinned Deno, a checkout owned by a service user, the built dists, and the right to bind 443. + +```bash +sudo apt install -y git unzip +curl -fsSLo /tmp/deno.zip https://github.com/denoland/deno/releases/download/v2.6.10/deno-x86_64-unknown-linux-gnu.zip +sudo unzip -o -d /usr/local/bin /tmp/deno.zip +sudo useradd --system --create-home --home /srv/relay relay +sudo -u relay git clone https://github.com/dimensionalOS/dimos /srv/relay/dimos +sudo -u relay sh -c 'cd /srv/relay/dimos/web && deno install --frozen && deno task -r build' +sudo mkdir /etc/relay +``` + +The pin is `DENO_VERSION` in [`dimos/utils/deno.py`](/dimos/utils/deno.py#L32); an arm64 VM downloads `deno-aarch64-unknown-linux-gnu.zip` instead. Put `auth.json` in `/etc/relay` owned by `relay` with mode 600, and use `install -o relay -g relay -m 600 "$RENEWED_LINEAGE"/fullchain.pem "$RENEWED_LINEAGE"/privkey.pem /etc/relay/ && systemctl restart dimos-relay` as the certbot deploy hook. `/etc/systemd/system/dimos-relay.service`: + +```ini +[Unit] +Description=dimos relay +After=network-online.target + +[Service] +User=relay +WorkingDirectory=/srv/relay/dimos/web +ExecStart=/usr/local/bin/deno run --frozen --allow-net --allow-read=/srv/relay/dimos/web,/etc/relay relay/main.ts --host 0.0.0.0 --port 443 --cockpit-dir cockpit/dist --sdk-dir sdk/dist --cert /etc/relay/fullchain.pem --key /etc/relay/privkey.pem --auth-file /etc/relay/auth.json +AmbientCapabilities=CAP_NET_BIND_SERVICE +Restart=on-failure + +[Install] +WantedBy=multi-user.target +``` + +`sudo systemctl enable --now dimos-relay`, then `journalctl -u dimos-relay -f` for the log. + +## 9. LAN without a domain + +On a LAN with no public name, sign your own certificate with mkcert. It creates a private CA and a certificate for the relay's address. The robot trusts the CA through `--relay-ca`. The auth file stays mandatory: the relay binds a non-loopback address only with certificate, key, and auth file together. The image and the compose file are the same as on a VM. Only the certificate step changes. `192.168.1.20` below is the relay machine's LAN address. + +```bash +sudo apt install -y mkcert libnss3-tools +mkcert -install +mkdir -p docker/relay/config +mkcert -cert-file docker/relay/config/fullchain.pem -key-file docker/relay/config/privkey.pem 192.168.1.20 +``` + +`libnss3-tools` is what lets `mkcert -install` add the CA to Firefox. Write `auth.json` as in section 4, give all three files to uid 1993 the same way, and start the relay as in section 5. + +The robot needs the CA bundle: its QUIC client trusts certifi's roots, not the system store. + +```bash +RELAY_KEY= uv run dimos --simulation run unitree-go2-agentic-cockpit --relay-url https://192.168.1.20 \ + --relay-ca "$(mkcert -CAROOT)/rootCA.pem" --robot-id go2-lab +``` + +Browsers are the catch. Firefox trusts the certificate for the page but drops QUIC through a root that is not built in. In `about:config` set `network.http.http3.disable_when_third_party_roots_found` to `false` and restart Firefox. Then open `https://192.168.1.20/` and enter the viewer token. Chromium refuses private roots for QUIC outright (`QUIC_CERT_ROOT_NOT_KNOWN`; no flag lifts it): the page loads over HTTPS and never connects. Chromium on a LAN needs a publicly trusted certificate, which a domain you own provides without any inbound port: point a name at the LAN address and use certbot's DNS challenge (`certbot certonly --manual --preferred-challenges dns -d relay.lan.example.com`). + +Without Docker, section 8's command works with the mkcert files in place of the certbot ones. + +## 10. Troubleshooting + +- **`auth_failed`.** The relay's message says `missing` (the client sent no secret: `RELAY_KEY` not in the robot's environment, no token stored in the browser) or `invalid` (wrong secret, or a robot key used with another robot id). It is terminal: the bridge stops and the page shows the token form. Fix the secret and start the client again. The relay log names the rejected robot id or viewer. +- **`accept failed: aborted by peer: the application or application protocol caused the connection to be closed during the handshake`**, repeating every 8 s, with no `viewer N authenticated` line. A browser is retrying and closing the connection itself after the TLS check: with a private CA, Firefox without the `about:config` pref from section 9, or Chromium. The page shows "Waiting for a robot to register..." meanwhile, which also covers a page still connecting; the status bar shows the real state. The robot and `/api/stats` are unaffected. +- **Certificate errors.** Check the VM clock (`timedatectl`). The robot fails with `CERTIFICATE_VERIFY_FAILED` when the relay serves `cert.pem` instead of `fullchain.pem` (browsers fetch the missing intermediate, OpenSSL does not) or when `--relay-ca` points at the wrong CA. +- **UDP 443 blocked.** HTTPS works, the page loads and stays "reconnecting", the robot's startup attempts time out (`relay startup connection attempt N failed`) and it gives up. Check both firewalls for UDP; `sudo ss -ulnp | grep 443` on the VM shows the published UDP port. +- **Robot behind NAT.** Fine: it dials out, and the QUIC session sends keepalives every 4 s. +- **Relay behind NAT** (a home server). Forward TCP 443 and UDP 443 to it. `/api/info` advertises the name the client dialed, so nothing else changes. +- **Restarts.** `docker compose restart relay` is always safe: robots re-register within seconds, pages reconnect on their own. A robot killed without a clean close holds its id for up to 30 s (the QUIC idle timeout); its restart waits that out. + +## 11. Checklist + +What a hosted relay is verified with; repeat it after changes to the relay, the image, or this guide. + +- [ ] Fresh Ubuntu VM with a domain, sections 2 to 5 top to bottom; `curl https://.../api/info` from a laptop. +- [ ] Replay robot on another, NATed network (`dimos --replay run unitree-go2 --relay-url ...`). +- [ ] Laptop on a third network, in Chromium and in Firefox: wrong token rejected, right token connects; robot picker with two robots; live map and video; teleop drives the replay robot. +- [ ] `docker compose restart relay` with the page open: the page reconnects, the robot re-registers. +- [ ] The deploy hook by hand: the relay restarts with the installed files; `sudo certbot renew --dry-run` passes. +- [ ] `/api/stats`: 401 without the token, 200 with it. +- [ ] Section 9 on a LAN: the robot registers with `--relay-ca`, Firefox with the pref connects, Chromium loads the page and never connects. diff --git a/docs/usage/web_sdk.md b/docs/usage/web_sdk.md index 8441e4873a..bc05bc062b 100644 --- a/docs/usage/web_sdk.md +++ b/docs/usage/web_sdk.md @@ -68,6 +68,8 @@ RELAY_KEY= uv run dimos run unitree-go2 --relay-url https://dimos-relay.exa The cockpit asks for the viewer token, keeps it in `localStorage`, and "log out" in the status bar forgets it. Your own page passes it to `connect({ url, token })`. `/api/stats` wants it as `Authorization: Bearer ` and drops its CORS header. A wrong key or token fails with `auth_failed` and neither client retries: fix the secret and restart. Edits to the file need a relay restart. +To host one on a VM with Docker and a Let's Encrypt certificate, or on a LAN with mkcert, follow [Relay hosting](/docs/usage/relay_hosting.md). + ## Your first page Create `ui/index.html`: diff --git a/web/README.md b/web/README.md index 80104e312d..5a45939144 100644 --- a/web/README.md +++ b/web/README.md @@ -118,6 +118,10 @@ until "log out"; the SDK takes `connect({url, token})`). A wrong secret fails wi which is terminal: neither client retries. `relay/auth.ts` compares in constant time and never logs a secret; edits to the file need a restart. +Hosting a relay on a VM (container image, compose file, certbot, firewall, the auth file) is +[docs/usage/relay_hosting.md](../docs/usage/relay_hosting.md); `docker/relay/` holds the Dockerfile +and the compose file. + ## Cockpit ```bash