fix(railway): restore runner and mount connectivity - #5485
Conversation
dosubot
Bot
added
the
size:M
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
mmabrouk
added
the
needs-review
ChatGPT Codex Connector
|
Important Review skippedNo new commits to review since the last review. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughRailway deployment now reuses or generates mount-store credentials, configures runners to bind on all interfaces, and applies shared HTTP 503 handling for mount-storage acceptance tests, with CI configured to require mount storage. ChangesRailway mount storage and runner configuration
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
🤖 The AI agent says: Verified the fix with Bash syntax checks and an isolated two-container network test: the loopback bind is unreachable from a peer, while the private-network bind returns the runner protocol health payload. @coderabbitai review |
|
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e8f2c5b4-fce7-4947-85bb-8ce23a29bfea
📒 Files selected for processing (4)
hosting/railway/oss/runner/Dockerfilehosting/railway/oss/scripts/configure.shhosting/railway/oss/scripts/deploy-from-images.shhosting/railway/oss/scripts/smoke.sh
| if railway ssh --service services --environment "$ENV_NAME" \ | ||
| python -c "import os, urllib.request; url = os.environ.get(\"AGENTA_RUNNER_INTERNAL_URL\", \"\").rstrip(\"/\") + \"/health\"; response = urllib.request.urlopen(url, timeout=5); raise SystemExit(0 if response.status == 200 else 1)" \ | ||
| >/dev/null; then |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -eu
: "${ENV_NAME:?}"
: "${RAILWAY_SSH_IDENTITY_FILE:?}"
command -v timeout
test -r "$RAILWAY_SSH_IDENTITY_FILE"
timeout 15s railway ssh \
--identity-file "$RAILWAY_SSH_IDENTITY_FILE" \
--service services \
--environment "$ENV_NAME" \
trueRepository: Agenta-AI/agenta
Length of output: 204
🏁 Script executed:
#!/usr/bin/env bash
set -eu
echo "== script excerpt =="
sed -n '110,155p' hosting/railway/oss/scripts/smoke.sh
echo
echo "== railway ssh references =="
rg -n "railway ssh|RAILWAY_SSH_IDENTITY_FILE|MAX_WAIT_SECONDS|ssh" hosting/railway/oss/scripts/smoke.sh .github hosting/railway 2>/dev/null | head -200
echo
echo "== local git status/stat =="
git diff --stat || true
git status --short || trueRepository: Agenta-AI/agenta
Length of output: 3690
🌐 Web query:
Railway CLI ssh --no-host-check --identity-file first use key registration timeout
💡 Result:
If you are experiencing issues with the Railway CLI SSH key registration or connectivity—such as timeouts, "Unauthorized" errors, or registration failures—there are several known behaviors and troubleshooting steps to resolve the issue. Regarding your specific mention of SSH key registration timeouts and flags: - Railway CLI SSH behavior: The Railway CLI's railway ssh command often requires local public key files to be present on disk (typically in ~/.ssh/ with a .pub extension) to register or identify your key [1]. It does not always automatically consult ssh-agent, which can lead to registration or authentication failures if your keys are only agent-resident [1]. - Manual SSH alternative: As of mid-2026, Railway supports standard native SSH connections [2][3]. If the CLI command fails, you can often bypass it by using the standard SSH client directly: ssh <user>@ssh.railway.com`` [2]. This method is generally more reliable as it natively supports ssh-agent, `~/.ssh/config` files, and standard SSH features [2]. - Troubleshooting "Unauthorized" or registration errors: - Ensure you are logged in via `railway login` [4]. Do not use a project-scoped `RAILWAY_TOKEN` for SSH, as SSH key management is not supported with these tokens [5][4]. Use an account-level session or a workspace-scoped `RAILWAY_API_TOKEN` [5][4]. - Verify your key is registered by running `railway ssh keys list` [6][4]. - If you are on a container that lacks a shell (e.g., `/bin/sh` or `/bin/bash`), `railway ssh` will fail, often returning "Connection closed" or "executable not found" errors [6]. Ensure your container base image includes a shell [6]. - For intermittent "deadline has elapsed" or connection timeouts, these are frequently attributed to network-specific issues or ISP restrictions on WebSocket connections, rather than the Railway platform itself [7]. If you are trying to force a specific identity file, you can use the `-i` flag with `railway ssh`, but ensure the corresponding public key file exists on your machine if you are relying on the CLI's internal registration logic [1][6][4]. For the most reliable results, especially when dealing with complex SSH configurations, prefer using the standard `ssh` client directly [2][3].
Citations:
- 1:
railway sshignoresssh-agent, fails when keys aren't on disk railwayapp/cli#870 - 2: https://station.railway.com/feedback/feedback-standard-ssh-b815eb3c
- 3: https://railway.com/changelog/2026-06-26-railway-over-ssh.md
- 4: https://docs.railway.com/cli/ssh
- 5: https://station.railway.com/questions/railway-ssh-fails-with-unauthorized-imme-4e1737f8
- 6: https://station.railway.com/questions/connection-to-ssh-railway-com-closed-552294aa
- 7: https://station.railway.com/questions/ssh-connection-intermittently-failing-wi-0d63f764
🏁 Script executed:
#!/usr/bin/env bash
set -eu
echo "== smoke script environment/defaults =="
sed -n '1,35p' hosting/railway/oss/scripts/smoke.sh
echo
echo "== workflow env around smoke runner =="
sed -n '90,120p' .github/workflows/43-railway-deploy.yml
echo
echo "== Railway CLI ssh documentation references in repo =="
rg -n "Railway|railway|first.*key|register|ssh" README.md .github hosting/railway 2>/dev/null | head -200Repository: Agenta-AI/agenta
Length of output: 26538
Bound and make the new railway ssh probe non-interactive.
The Python timeout=5 only applies after the SSH session is established; this can still block the loop indefinitely if the Railway CLI hangs during SSH connection/authentication or first-use key registration. Use a non-interactive CI key and wrap the invocation in a bounded timeout.
Railway Preview Environment
|
5eae8a7 to
d326b91
Compare
dosubot
Bot
added
size:XS
|
🤖 The AI agent says: Amended after diagnosing the failed preview workflow:
Focused shell syntax, whitespace, pre-commit, pre-push, and remote-SHA checks pass. The replacement Railway preview run is now in progress. @coderabbitai review |
|
✅ Action performedReview finished.
|
|
🤖 The AI agent says: Final verification is complete. The replacement Railway preview workflow passed end to end: image builds, environment setup, deploy, readiness, and every API, SDK, services, and web unit/integration/acceptance suite are green. Workflow: https://github.com/Agenta-AI/agenta/actions/runs/30128388088 The PR remains |
mmabrouk
added
implementing
dosubot
Bot
added
size:M
mmabrouk
changed the title
mmabrouk
added
needs-review
|
🤖 The AI agent says: I amended this PR after the Railway preview exposed a second deployment failure:
Local formatting, linting, shell syntax, YAML parsing, and collection of all 34 affected mount tests pass. The new Railway preview run is the end-to-end check. |
|
🤖 The AI agent says: @coderabbitai review |
|
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 291c4d10-fd0c-4f66-b023-fd16a701d429
📒 Files selected for processing (7)
.github/workflows/44-railway-tests.ymlapi/oss/tests/pytest/acceptance/mounts/test_agent_mounts_basics.pyapi/oss/tests/pytest/acceptance/mounts/test_mounts_basics.pyapi/oss/tests/pytest/acceptance/mounts/test_mounts_injection.pyapi/oss/tests/pytest/utils/mounts.pyhosting/railway/oss/README.mdhosting/railway/oss/scripts/configure.sh
| api_variables="$(railway_call variable list --json --service api --environment "$ENV_NAME")" | ||
| if ! jq -e 'type == "object"' >/dev/null 2>&1 <<<"$api_variables"; then | ||
| printf "Unable to read existing mount storage configuration from the Railway API service.\n" >&2 | ||
| return 1 | ||
| fi | ||
|
|
||
| if [ -z "$AGENTA_STORE_ACCESS_KEY" ]; then | ||
| AGENTA_STORE_ACCESS_KEY="$(_store_variable_from_json "$api_variables" AGENTA_STORE_ACCESS_KEY)" | ||
| fi | ||
| if [ -z "$AGENTA_STORE_SECRET_KEY" ]; then | ||
| AGENTA_STORE_SECRET_KEY="$(_store_variable_from_json "$api_variables" AGENTA_STORE_SECRET_KEY)" | ||
| fi | ||
| if [ -z "$AGENTA_STORE_SIGNING_KEY" ]; then | ||
| AGENTA_STORE_SIGNING_KEY="$(_store_variable_from_json "$api_variables" AGENTA_STORE_SIGNING_KEY)" | ||
| fi | ||
| if [ -z "$AGENTA_STORE_JWT_PRIVATE_KEY" ]; then | ||
| AGENTA_STORE_JWT_PRIVATE_KEY="$(_store_variable_from_json "$api_variables" AGENTA_STORE_JWT_PRIVATE_KEY)" | ||
| fi | ||
|
|
||
| AGENTA_STORE_ACCESS_KEY="${AGENTA_STORE_ACCESS_KEY:-$(openssl rand -hex 10)}" | ||
| AGENTA_STORE_SECRET_KEY="${AGENTA_STORE_SECRET_KEY:-$(openssl rand -hex 32)}" | ||
| AGENTA_STORE_SIGNING_KEY="${AGENTA_STORE_SIGNING_KEY:-$(openssl rand -base64 32)}" | ||
| AGENTA_STORE_JWT_PRIVATE_KEY="${AGENTA_STORE_JWT_PRIVATE_KEY:-$(openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 2>/dev/null)}" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)hosting/railway/oss/(scripts/configure\.sh|README\.md)$|configure\.sh|README\.md' || true
echo
echo "== configure.sh relevant section =="
if [ -f hosting/railway/oss/scripts/configure.sh ]; then
nl -ba hosting/railway/oss/scripts/configure.sh | sed -n '1,140p'
fi
echo
echo "== README relevant section =="
if [ -f hosting/railway/oss/README.md ]; then
nl -ba hosting/railway/oss/README.md | sed -n '1,100p'
fi
echo
echo "== railway variable access helper usages =="
rg -n "railway_call variable|_store_variable_from_json|AGENTA_STORE_.*KEY|AGENTA_STORE_JWT_PRIVATE_KEY|variable list --json" hosting/railway/oss -S || trueRepository: Agenta-AI/agenta
Length of output: 10572
🌐 Web query:
Railway variables sealed variables CLI variable list omitted environment variables documentation
💡 Result:
In Railway, environment variables can be managed via the dashboard UI, the CLI, and the public GraphQL API [1][2][3]. Sealed Variables Sealed variables are a security feature designed to prevent sensitive values from being read back after they are set [4][5]. When a variable is sealed: - The value remains available to your application at runtime and during the build process as a standard environment variable [4][5]. - The value is never visible in the Railway dashboard UI [4][5]. - The value cannot be retrieved via the Public API [4][5]. - There is currently no CLI command or API mutation to create or seal a variable; this action must be performed manually in the Railway dashboard UI using the variable's 3-dot menu [6]. CLI Variable Management The Railway CLI provides command-line access to manage variables for your services [2][7]. You can use the railway variable (or railway var / railway vars) command to interact with your variables [2]: - List variables: railway variable list [2][8] - Use -s or --service to target a specific service [2][9]. - Use -e or --environment to target a specific environment [2][9]. - Use --kv to output in KEY=VALUE format [2][8]. - Use --json to output in JSON format [2][8]. - Set a variable: railway variable set KEY=VALUE [2][7] - Delete a variable: railway variable delete KEY [2][7] Note that when listing variables via the CLI, if a variable is "sealed," its value will not be returned or displayed, ensuring the security of the secret [8]. Additionally, ensure you are authenticated and have linked your project to use these commands [1].
Citations:
- 1: https://docs.railway.com/variables
- 2: https://docs.railway.com/cli/variable
- 3: https://docs.railway.com/integrations/api/manage-variables
- 4: https://docs.railway.com/guides/managing-secrets-on-railway
- 5: https://station.railway.com/questions/what-is-the-purpose-of-sealed-variables-dcea7688
- 6: https://station.railway.com/questions/api-or-cli-for-sealed-variables-24e8f2b8
- 7: https://docs.railway.com/cli
- 8: https://github.com/railwayapp/cli/blob/v4.58.0/src/commands/variable.rs
- 9: https://docs.railway.com/cli/global-options
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate candidate files =="
printf '%s\n' "hosting/railway/oss/scripts/configure.sh" "hosting/railway/oss/README.md"
if [ -f hosting/railway/oss/scripts/configure.sh ]; then
python3 - <<'PY'
from pathlib import Path
p=Path('hosting/railway/oss/scripts/configure.sh')
for i, line in enumerate(p.read_text().splitlines(), 1):
if 1 <= i <= 140:
print(f"{i:5d}\t{line}")
PY
fi
echo
if [ -f hosting/railway/oss/README.md ]; then
python3 - <<'PY'
from pathlib import Path
p=Path('hosting/railway/oss/README.md')
for i, line in enumerate(p.read_text().splitlines(), 1):
if 1 <= i <= 100:
print(f"{i:5d}\t{line}")
PY
fi
echo
echo "== search credential handling =="
python3 - <<'PY'
import subprocess
patterns = [
"railway_call variable",
"_store_variable_from_json",
"AGENTA_STORE_ACCESS_KEY",
"AGENTA_STORE_SECRET_KEY",
"AGENTA_STORE_SIGNING_KEY",
"AGENTA_STORE_JWT_PRIVATE_KEY",
"variable list --json"
]
for pat in patterns:
print(f"\n-- {pat!r} --")
subprocess.run(["grep", "-n", "-R", "-S", pat, "hosting/railway/oss"], text=True)
PYRepository: Agenta-AI/agenta
Length of output: 12540
🌐 Web query:
Railway docs variables sealed variables value cannot be retrieved via the Public API CLI variable list omit sealed values
💡 Result:
In Railway, sealed variables are intentionally designed to be write-only for security reasons [1][2]. Because of this, their values cannot be retrieved or displayed via the Public API or the Railway CLI [1][2]. When you list variables using the Railway CLI (e.g., via railway variable list or railway run), sealed variables are omitted from the output [1][2]. Similarly, the Public API will not return the values of these variables in any response [2][3]. Railway injects these values directly into your builds and deployments at runtime, ensuring they remain protected from visibility in the dashboard UI, API responses, or CLI commands [1][2][4]. Because sealing is a permanent, one-way security measure, you cannot unseal a variable once it has been set [1][2]. If you need to access a secret value locally for development that is also used as a sealed variable in production, the recommended practice is to maintain a separate, non-sealed variable for your local development environment [1][2].
Citations:
- 1: https://docs.railway.com/guides/managing-secrets-on-railway
- 2: https://docs.railway.com/variables
- 3: https://station.railway.com/feedback/sealed-variables-c1317d54
- 4: https://station.railway.com/questions/what-is-the-purpose-of-sealed-variables-dcea7688
Preserve sealed mount-store credentials instead of regenerating them.
Railway does not return sealed variable values from the CLI/API, so _store_variable_from_json treats them as absent and the script generates replacements. That violates the documented reuse behavior and can rotate credentials for an existing deployment. Distinguish “sealed exists” from “missing” by requiring explicit overrides or persisting credentials outside Railway for future configure.sh runs.
📍 Affects 2 files
hosting/railway/oss/scripts/configure.sh#L60-L82(this comment)hosting/railway/oss/README.md#L63-L66
|
Amendment verification update:
— Codex |
1 participant
Context
Railway previews had two failures in the agent workflow. Agent requests could not reach the runner because it listened only on its container loopback address. Mount file requests returned
503 Service Unavailablebecause the deployment wrote empty SeaweedFS access and secret keys to every service.The Railway health checks did not catch either failure. The runner could answer its own container health check, and the API acceptance tests skipped mount tests when storage returned 503.
Changes
The Railway runner now binds to
0.0.0.0, soservicescan reach it through Railway private networking. The existing sharedAGENTA_RUNNER_TOKENstill authenticates those requests.configure.shnow treats the API service as the durable copy of the mount-storage credentials:Before this change, a fresh preview stored empty access and secret keys, and every configure run rotated the signing and JWT keys. After this change, a fresh preview configures working storage automatically, while later redeployments keep the same credentials.
The Railway API test job now requires mount storage. A 503 fails the mount acceptance tests in Railway instead of silently skipping them. Other environments can still skip these tests when mount storage is intentionally absent.
How to review
hosting/railway/oss/scripts/configure.sh. Review the credential precedence and confirm resolution happens after Railway linking but before service variables are written..github/workflows/44-railway-tests.yml.deploy-from-images.shconfigures credentials before restarting SeaweedFS and deploying API, services, and runner.Verification
ruff format .andruff check --fix .inapibash -nfor the Railway configure, image-deploy, and smoke scripts.github/workflows/44-railway-tests.ymlas YAMLgit diff --checkThe Railway preview workflow will provide the end-to-end verification after this revision deploys.
What to QA