Skip to content

fix(railway): restore runner and mount connectivity #5485

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mmabrouk wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(railway): restore runner and mount connectivity #5485

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d326b91
fix(railway): expose runner on the private network
mmabrouk Jul 24, 2026
1771252
fix(railway): persist mount storage credentials
mmabrouk Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/44-railway-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,7 @@ jobs:
AGENTA_LICENSE: oss
AGENTA_API_URL: ${{ needs.prepare.outputs.agenta_api_url }}
AGENTA_AUTH_KEY: ${{ secrets.AGENTA_TEST_OSS_AUTH_KEY }}
AGENTA_TEST_REQUIRE_MOUNT_STORAGE: "true"
steps:
- uses: actions/checkout@v6

Expand Down
9 changes: 4 additions & 5 deletions api/oss/tests/pytest/acceptance/mounts/test_agent_mounts_basics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
"""Acceptance tests for artifact-scoped agent mounts.

Query-only tests run without an object store. Credential-signing tests skip when
the running API has no mount storage backend configured. Sign verifies the
Query-only tests run without an object store. Credential-signing tests fail when
the environment requires mount storage and skip otherwise. Sign verifies the
artifact exists in the project (404 otherwise), so signing tests create a real
workflow first.
"""

from uuid import uuid4

import pytest
from oss.tests.pytest.utils.mounts import skip_if_mount_storage_unavailable


def _create_workflow(authed_api):
Expand All @@ -35,8 +35,7 @@ def _sign_agent_mount(authed_api, artifact_id, *, name="default"):
"/mounts/agents/sign",
params={"artifact_id": artifact_id, "name": name},
)
if response.status_code == 503:
pytest.skip("Mount storage backend not configured in this environment")
skip_if_mount_storage_unavailable(response)
return response


Expand Down
19 changes: 7 additions & 12 deletions api/oss/tests/pytest/acceptance/mounts/test_mounts_basics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

from uuid import uuid4

from oss.tests.pytest.utils.mounts import skip_if_mount_storage_unavailable


def _mount_payload(*, name=None, session_id=None):
# Storage location is derived server-side (bucket from env, key = project_id/mount_id);
Expand Down Expand Up @@ -231,8 +233,8 @@ def test_session_query_does_not_return_other_session_mounts(self, authed_api):
# File ops (durable store contents)
#
# Require a configured object store (AGENTA_STORE_* → SeaweedFS in the dev
# stack). A 503 means the backend isn't configured in this env — skip rather
# than fail so the suite stays green where no store is wired.
# stack). Environments that require mount storage fail on a 503; other
# environments skip these checks.
# ---------------------------------------------------------------------------


Expand All @@ -243,13 +245,6 @@ def _create_mount(authed_api):
return resp.json()["mount"]["id"]


def _skip_if_no_store(resp):
import pytest

if resp.status_code == 503:
pytest.skip("Mount storage backend not configured in this environment")


class TestMountFileOps:
def test_write_read_list_delete_roundtrip(self, authed_api):
mount_id = _create_mount(authed_api)
Expand All @@ -260,7 +255,7 @@ def test_write_read_list_delete_roundtrip(self, authed_api):
params={"path": "notes.txt"},
data=b"hello world",
)
_skip_if_no_store(write)
skip_if_mount_storage_unavailable(write)
assert write.status_code == 200, write.text
assert write.json()["path"] == "notes.txt"

Expand Down Expand Up @@ -291,7 +286,7 @@ def test_create_folder_upload_read_cascade_delete(self, authed_api):
folder = authed_api(
"POST", f"/mounts/{mount_id}/files/folder", params={"path": "workspace"}
)
_skip_if_no_store(folder)
skip_if_mount_storage_unavailable(folder)
assert folder.status_code == 200, folder.text
assert folder.json()["path"] == "workspace"

Expand Down Expand Up @@ -342,7 +337,7 @@ def test_upload_falls_back_to_filename(self, authed_api):
f"/mounts/{mount_id}/files/upload",
files={"file": ("report.txt", b"content", "text/plain")},
)
_skip_if_no_store(upload)
skip_if_mount_storage_unavailable(upload)
assert upload.status_code == 200, upload.text
assert upload.json()["path"] == "report.txt"

Expand Down
22 changes: 9 additions & 13 deletions api/oss/tests/pytest/acceptance/mounts/test_mounts_injection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,16 @@
(the cross-turn persistence contract, exercised at the store level).

Requires a configured object store with a working STS endpoint (AGENTA_STORE_* →
SeaweedFS in the dev stack). A 503 means no store / STS — skip rather than fail.
SeaweedFS in the dev stack). Environments that require mount storage fail on a 503;
other environments skip these checks.

The full runner round-trip (geesefs mount inside a live run, file visible across two turns)
is a manual/live check on the dev stack; see docs/designs/sessions/mounts/extend/tasks.md M10.
"""

from uuid import uuid4

import pytest


def _skip_if_no_store(resp):
if resp.status_code == 503:
pytest.skip("Mount storage / STS backend not configured in this environment")
from oss.tests.pytest.utils.mounts import skip_if_mount_storage_unavailable


def _sign_session(authed_api, session_id):
Expand All @@ -36,7 +32,7 @@ class TestSessionMountSign:
def test_bind_and_sign_returns_scoped_credentials(self, authed_api):
session_id = f"session-{uuid4().hex[:8]}"
resp = _sign_session(authed_api, session_id)
_skip_if_no_store(resp)
skip_if_mount_storage_unavailable(resp)
assert resp.status_code == 200, resp.text

body = resp.json()
Expand All @@ -59,7 +55,7 @@ def test_bind_and_sign_returns_scoped_credentials(self, authed_api):
def test_bind_is_idempotent_same_session_same_mount(self, authed_api):
session_id = f"session-{uuid4().hex[:8]}"
first = _sign_session(authed_api, session_id)
_skip_if_no_store(first)
skip_if_mount_storage_unavailable(first)
assert first.status_code == 200, first.text

second = _sign_session(authed_api, session_id)
Expand All @@ -74,7 +70,7 @@ def test_bind_is_idempotent_same_session_same_mount(self, authed_api):

def test_different_sessions_get_different_prefixes(self, authed_api):
a = _sign_session(authed_api, f"session-{uuid4().hex[:8]}")
_skip_if_no_store(a)
skip_if_mount_storage_unavailable(a)
assert a.status_code == 200, a.text
b = _sign_session(authed_api, f"session-{uuid4().hex[:8]}")
assert b.status_code == 200, b.text
Expand Down Expand Up @@ -103,7 +99,7 @@ def test_sign_existing_mount(self, authed_api):
mount = create.json()["mount"]

resp = authed_api("POST", f"/mounts/{mount['id']}/sign")
_skip_if_no_store(resp)
skip_if_mount_storage_unavailable(resp)
assert resp.status_code == 200, resp.text

creds = resp.json()["credentials"]
Expand All @@ -120,7 +116,7 @@ class TestDurablePrefixSurvivesRebind:
def test_file_survives_rebind(self, authed_api):
session_id = f"session-{uuid4().hex[:8]}"
first = _sign_session(authed_api, session_id)
_skip_if_no_store(first)
skip_if_mount_storage_unavailable(first)
assert first.status_code == 200, first.text
mount_id = first.json()["mount"]["id"]

Expand All @@ -131,7 +127,7 @@ def test_file_survives_rebind(self, authed_api):
params={"path": "turn1.txt"},
data=b"written in turn 1",
)
_skip_if_no_store(write)
skip_if_mount_storage_unavailable(write)
assert write.status_code == 200, write.text

# Re-bind (a fresh "turn"): same mount, and the file is still readable.
Expand Down
13 changes: 13 additions & 0 deletions api/oss/tests/pytest/utils/mounts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import os

import pytest


def skip_if_mount_storage_unavailable(response):
if response.status_code != 503:
return

if os.getenv("AGENTA_TEST_REQUIRE_MOUNT_STORAGE", "").lower() == "true":
pytest.fail(f"Mount storage is required but unavailable: {response.text}")

pytest.skip("Mount storage backend not configured in this environment")
5 changes: 5 additions & 0 deletions hosting/railway/oss/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,11 @@ export AGENTA_RUNNER_TOKEN="$(openssl rand -hex 32)"
export POSTGRES_PASSWORD="$(openssl rand -hex 24)"
```

`configure.sh` creates the bundled mount store credentials on a fresh Railway
project and reuses them on later deployments. You can provide
`AGENTA_STORE_ACCESS_KEY`, `AGENTA_STORE_SECRET_KEY`,
`AGENTA_STORE_SIGNING_KEY`, and `AGENTA_STORE_JWT_PRIVATE_KEY` to override them.

## Quick Start

```bash
Expand Down
1 change: 1 addition & 0 deletions hosting/railway/oss/runner/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

FROM ghcr.io/agenta-ai/agenta-runner:latest

ENV AGENTA_RUNNER_HOST=0.0.0.0
ENV AGENTA_RUNNER_PORT=8765

CMD ["node_modules/.bin/tsx", "src/server.ts"]
49 changes: 46 additions & 3 deletions hosting/railway/oss/scripts/configure.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-}"
AGENTA_STORE_ACCESS_KEY="${AGENTA_STORE_ACCESS_KEY:-}"
AGENTA_STORE_SECRET_KEY="${AGENTA_STORE_SECRET_KEY:-}"
AGENTA_STORE_BUCKET="${AGENTA_STORE_BUCKET:-agenta-store}"
AGENTA_STORE_SIGNING_KEY="${AGENTA_STORE_SIGNING_KEY:-$(openssl rand -base64 32)}"
AGENTA_STORE_SIGNING_KEY="${AGENTA_STORE_SIGNING_KEY:-}"
# RSA key the API signs its store web-identity token with; the bundled SeaweedFS verifies it
# against the API's JWKS. Generated once per configure run if unset (single-replica Railway).
AGENTA_STORE_JWT_PRIVATE_KEY="${AGENTA_STORE_JWT_PRIVATE_KEY:-$(openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 2>/dev/null)}"
# against the API's JWKS.
AGENTA_STORE_JWT_PRIVATE_KEY="${AGENTA_STORE_JWT_PRIVATE_KEY:-}"

# Populated by resolve_railway_ids() after `railway link`. Used by the GraphQL
# variableCollectionUpsert path; left empty -> upsert_service_vars falls back
Expand All @@ -46,6 +46,44 @@ resolve_postgres_password() {
fi
}

_store_variable_from_json() {
local variables_json="$1"
local key="$2"

jq -r --arg key "$key" \
'.[$key] // empty | select(type == "string" and length > 0)' \
<<<"$variables_json"
}

resolve_store_configuration() {
local api_variables
api_variables="$(railway_call variable list --json --service api --environment "$ENV_NAME")"
if ! jq -e 'type == "object"' >/dev/null 2>&1 <<<"$api_variables"; then
printf "Unable to read existing mount storage configuration from the Railway API service.\n" >&2
return 1
fi

if [ -z "$AGENTA_STORE_ACCESS_KEY" ]; then
AGENTA_STORE_ACCESS_KEY="$(_store_variable_from_json "$api_variables" AGENTA_STORE_ACCESS_KEY)"
fi
if [ -z "$AGENTA_STORE_SECRET_KEY" ]; then
AGENTA_STORE_SECRET_KEY="$(_store_variable_from_json "$api_variables" AGENTA_STORE_SECRET_KEY)"
fi
if [ -z "$AGENTA_STORE_SIGNING_KEY" ]; then
AGENTA_STORE_SIGNING_KEY="$(_store_variable_from_json "$api_variables" AGENTA_STORE_SIGNING_KEY)"
fi
if [ -z "$AGENTA_STORE_JWT_PRIVATE_KEY" ]; then
AGENTA_STORE_JWT_PRIVATE_KEY="$(_store_variable_from_json "$api_variables" AGENTA_STORE_JWT_PRIVATE_KEY)"
fi

AGENTA_STORE_ACCESS_KEY="${AGENTA_STORE_ACCESS_KEY:-$(openssl rand -hex 10)}"
AGENTA_STORE_SECRET_KEY="${AGENTA_STORE_SECRET_KEY:-$(openssl rand -hex 32)}"
AGENTA_STORE_SIGNING_KEY="${AGENTA_STORE_SIGNING_KEY:-$(openssl rand -base64 32)}"
AGENTA_STORE_JWT_PRIVATE_KEY="${AGENTA_STORE_JWT_PRIVATE_KEY:-$(openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 2>/dev/null)}"
Comment on lines +60 to +82

@coderabbitai coderabbitai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== files =="
git ls-files | rg '(^|/)hosting/railway/oss/(scripts/configure\.sh|README\.md)$|configure\.sh|README\.md' || true

echo
echo "== configure.sh relevant section =="
if [ -f hosting/railway/oss/scripts/configure.sh ]; then
  nl -ba hosting/railway/oss/scripts/configure.sh | sed -n '1,140p'
fi

echo
echo "== README relevant section =="
if [ -f hosting/railway/oss/README.md ]; then
  nl -ba hosting/railway/oss/README.md | sed -n '1,100p'
fi

echo
echo "== railway variable access helper usages =="
rg -n "railway_call variable|_store_variable_from_json|AGENTA_STORE_.*KEY|AGENTA_STORE_JWT_PRIVATE_KEY|variable list --json" hosting/railway/oss -S || true

Repository: Agenta-AI/agenta

Length of output: 10572

🌐 Web query:

Railway variables sealed variables CLI variable list omitted environment variables documentation

💡 Result:

In Railway, environment variables can be managed via the dashboard UI, the CLI, and the public GraphQL API [1][2][3]. Sealed Variables Sealed variables are a security feature designed to prevent sensitive values from being read back after they are set [4][5]. When a variable is sealed: - The value remains available to your application at runtime and during the build process as a standard environment variable [4][5]. - The value is never visible in the Railway dashboard UI [4][5]. - The value cannot be retrieved via the Public API [4][5]. - There is currently no CLI command or API mutation to create or seal a variable; this action must be performed manually in the Railway dashboard UI using the variable's 3-dot menu [6]. CLI Variable Management The Railway CLI provides command-line access to manage variables for your services [2][7]. You can use the railway variable (or railway var / railway vars) command to interact with your variables [2]: - List variables: railway variable list [2][8] - Use -s or --service to target a specific service [2][9]. - Use -e or --environment to target a specific environment [2][9]. - Use --kv to output in KEY=VALUE format [2][8]. - Use --json to output in JSON format [2][8]. - Set a variable: railway variable set KEY=VALUE [2][7] - Delete a variable: railway variable delete KEY [2][7] Note that when listing variables via the CLI, if a variable is "sealed," its value will not be returned or displayed, ensuring the security of the secret [8]. Additionally, ensure you are authenticated and have linked your project to use these commands [1].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate candidate files =="
printf '%s\n' "hosting/railway/oss/scripts/configure.sh" "hosting/railway/oss/README.md"
if [ -f hosting/railway/oss/scripts/configure.sh ]; then
  python3 - <<'PY'
from pathlib import Path
p=Path('hosting/railway/oss/scripts/configure.sh')
for i, line in enumerate(p.read_text().splitlines(), 1):
    if 1 <= i <= 140:
        print(f"{i:5d}\t{line}")
PY
fi
echo
if [ -f hosting/railway/oss/README.md ]; then
  python3 - <<'PY'
from pathlib import Path
p=Path('hosting/railway/oss/README.md')
for i, line in enumerate(p.read_text().splitlines(), 1):
    if 1 <= i <= 100:
        print(f"{i:5d}\t{line}")
PY
fi

echo
echo "== search credential handling =="
python3 - <<'PY'
import subprocess
patterns = [
    "railway_call variable",
    "_store_variable_from_json",
    "AGENTA_STORE_ACCESS_KEY",
    "AGENTA_STORE_SECRET_KEY",
    "AGENTA_STORE_SIGNING_KEY",
    "AGENTA_STORE_JWT_PRIVATE_KEY",
    "variable list --json"
]
for pat in patterns:
    print(f"\n-- {pat!r} --")
    subprocess.run(["grep", "-n", "-R", "-S", pat, "hosting/railway/oss"], text=True)
PY

Repository: Agenta-AI/agenta

Length of output: 12540

🌐 Web query:

Railway docs variables sealed variables value cannot be retrieved via the Public API CLI variable list omit sealed values

💡 Result:

In Railway, sealed variables are intentionally designed to be write-only for security reasons [1][2]. Because of this, their values cannot be retrieved or displayed via the Public API or the Railway CLI [1][2]. When you list variables using the Railway CLI (e.g., via railway variable list or railway run), sealed variables are omitted from the output [1][2]. Similarly, the Public API will not return the values of these variables in any response [2][3]. Railway injects these values directly into your builds and deployments at runtime, ensuring they remain protected from visibility in the dashboard UI, API responses, or CLI commands [1][2][4]. Because sealing is a permanent, one-way security measure, you cannot unseal a variable once it has been set [1][2]. If you need to access a secret value locally for development that is also used as a sealed variable in production, the recommended practice is to maintain a separate, non-sealed variable for your local development environment [1][2].

Citations:

Preserve sealed mount-store credentials instead of regenerating them.

Railway does not return sealed variable values from the CLI/API, so _store_variable_from_json treats them as absent and the script generates replacements. That violates the documented reuse behavior and can rotate credentials for an existing deployment. Distinguish “sealed exists” from “missing” by requiring explicit overrides or persisting credentials outside Railway for future configure.sh runs.

📍 Affects 2 files
  • hosting/railway/oss/scripts/configure.sh#L60-L82 (this comment)
  • hosting/railway/oss/README.md#L63-L66


printf "Mount storage credentials are configured for this Railway environment.\n"
}

require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
printf "Missing required command: %s\n" "$1" >&2
Expand Down Expand Up @@ -263,6 +301,10 @@ main() {
# the project/environment/services exist and are linked).
resolve_railway_ids

# The API service is the canonical durable copy. Explicit operator values win;
# otherwise reuse the current values and generate only what a fresh project lacks.
resolve_store_configuration

railway_call domain --service gateway --json >/dev/null 2>&1 || true

local public_domain_ref
Expand Down Expand Up @@ -365,6 +407,7 @@ main() {
"DAYTONA_API_KEY=${DAYTONA_API_KEY:-}"

set_vars runner \
AGENTA_RUNNER_HOST=0.0.0.0 \
AGENTA_RUNNER_PORT=8765 \
AGENTA_RUNNER_TOKEN="$AGENTA_RUNNER_TOKEN" \
AGENTA_RUNNER_ENABLED_SANDBOX_PROVIDERS="${AGENTA_RUNNER_ENABLED_SANDBOX_PROVIDERS:-local}" \
Expand Down
1 change: 1 addition & 0 deletions hosting/railway/oss/scripts/deploy-from-images.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ render_runner_wrapper() {
cat > "$dir/Dockerfile" <<EOF
FROM ${AGENTA_RUNNER_IMAGE}

ENV AGENTA_RUNNER_HOST=0.0.0.0
ENV AGENTA_RUNNER_PORT=8765

CMD ["node_modules/.bin/tsx", "src/server.ts"]
Expand Down
Loading