Skip to content

Latest commit

 

History

History
118 lines (108 loc) · 9.82 KB

AGENTS.md

File metadata and controls

118 lines (108 loc) · 9.82 KB

AGENTS.md

The GlueOps codespace: the devcontainer image (ghcr.io/glueops/codespaces) with the tools to run the GlueOps platform, and the VM setup that runs it as a cloud development environment (CDE) created from Slack.

Repo map

  • .devcontainer/Dockerfile: the codespace image. .devcontainer/tools/ holds the tools installed into it (cde-boot, cde-init, cde-trust-serve-web, captain_utils, gluekube_ssh, ...), including the tofu wrapper (tools/tofu.sh; the real binary is /usr/local/libexec/tofu). .devcontainer/misc-configs/tofurc becomes ~/.tofurc (README "OpenTofu mirror").
  • developer-setup.sh: VM setup, run at golden-image build (Packer) and by the break-glass curl setup.glueops.dev | bash. It embeds the VM's scripts and systemd units as heredocs, so it must stay a single self-contained file: codespace-ensure, codespace-tunnel, codespace-ttyd, dev, /usr/local/lib/codespace/common.sh, and the codespace-ensure.{service,timer}, codespace-tunnel@.service (instances editor, shell, vm) and codespace-ttyd@.service (instances shell, vm) units. README "The codespace VM" explains them, and its "Browser shells" section has the files the Slack bot writes.
  • qemu.pkr.hcl, os-setup-start.sh, os-setup-finish.sh: the Packer VM image.
  • hack/: test runners (hack/test.sh runs them all) and hack/vm-recovery-test.sh (manual, against a real VM).
  • tests/vm/: behaviour tests and stubs for the VM scripts. App tests live under .devcontainer/tests/<app>/.

Build and test (in Docker)

  • hack/test-codespace-vm.sh: the VM scripts and units, inside a pinned debian:13 container (needs docker).
  • hack/test-captain-utils.sh, hack/test-gluekube-ssh.sh: the image's tools.
  • hack/test-app-chart-preflight.sh: app-chart-preflight against stubbed kubectl and argocd, no cluster. Runs on the host and needs bash, jq, yq (mikefarah v4), python3 (the stub Argo CD server), curl, diff and timeout.
  • hack/test.sh: all of the above (CODESPACE_VM_TEST_DOCKER=1 to include the VM suite).
  • Each suite has its own workflow under .github/workflows/ with path triggers.

Rules

  • Each embedded file in developer-setup.sh has its own quoted heredoc delimiter (<<'CODESPACE_..._EOF'), never <<-, and the test runner extracts files by those delimiters. Nothing it runs at setup time may read stdin (curl | bash feeds the script on stdin).
  • Units that run after cloud-final must be WantedBy=cloud-init.target, never multi-user.target: cloud-final is ordered after multi-user.target, and the cycle makes systemd drop the job at boot. The VM test suite checks this. codespace-ttyd@<id> is wanted by codespace-tunnel@<id> instead, which is wanted by cloud-init.target.
  • The tokens (/etc/glueops/cde_token, /etc/glueops/shell_token) never go on a command line, into the environment, into logs or into a URL the VM prints. code serve-web prints the editor's on stdout, so its stdout is discarded. One exception: the shell token on ttyd's -c (ttyd has no credential-file option), so ps and systemctl status codespace-ttyd@<id> show it; hints point to journalctl, never systemctl status. journald stores the command line with every line a process logs, so ttyd's output is discarded too (the script's own lines, before it execs ttyd, still reach the journal). Leak checks look for each token and for the base64 of vscode:<shell token>, which is what ttyd prints.
  • Names: each thing the VM serves through sish has a short id (editor, shell, vm, listed in CODESPACE_ACCESS in common.sh) used unchanged in machine names: units (codespace-tunnel@<id>, codespace-ttyd@<id>), script arguments, subdomains (<id>-<hostname>; the editor keeps the bare hostname) and the common.sh helpers (codespace_bind, codespace_url, codespace_port, codespace_token_file). People see its label: "Editor", "Codespace shell", "VM shell". In those names and labels, "codespace" means the container, "VM" the host, and "terminal" only VS Code's own; a bare "tunnel" is a sish tunnel, the editor's other mode is "VS Code tunnel". Something new gets an id and a label and follows the same names.
  • packer-qemu.yml is triggered by workflow_run, so it always runs the default branch's copy: a tag builds its own qemu.pkr.hcl with main's workflow (runner, env, PKR_VAR_*). A pre-release can't test edits to that workflow, and qemu.pkr.hcl's defaults must work under main's.
  • Conventional Commits, release-please friendly.

Decisions

Settled; reviewers shouldn't re-raise these.

  • VM and boot changes may be breaking. Existing VMs are not migrated: people create a new codespace from Slack.
  • No memory cap on the container and no swap file. The OOM kills come from other workloads; recovery handles them.
  • The editor runs as the container's main process, supervised by Docker's restart policy plus the per-minute codespace-ensure. A crash of that process also ends shells and jobs inside the container. That's accepted.
  • No automatic recreate of a container that won't start; dev status points to dev reset.
  • The home and /nix volumes keep the home contents from the image tag they were first created with, even if a newer tag later runs on the same VM. Upgrading means a new VM. Volumes are not a backup.
  • The browser editor's VS Code server is pinned (serve-web --commit-id) to the one baked into the image: a VM gets no VS Code updates in place, including security releases, until a new VM. In exchange there's no ~700 MB download on a browser connect and the workspace-trust shim always applies.
  • VS Code is one version, ARG VERSION_VSCODE in the Dockerfile (code package, CLI, and the baked server via the CLI's commit). New images get a newer VS Code only when Renovate bumps it; renovate.json exempts it from the org's 30-day wait so security releases reach new images quickly.
  • The browser editor runs with --disable-telemetry. code tunnel has no such flag; its users' own VS Code settings apply.
  • Credentials kept in the home volume survive a container recreate. dev reset removes them.
  • cloud-init uses manual_cache_clean: true (os-setup-finish.sh): it discovers its datasource on first boot only, so a stop/start on libvirt/Proxmox (seed attached only at first boot) doesn't re-probe for ~6 min or regenerate host keys. Accepted: a VM copied from an already-booted disk isn't re-initialised (cloud-init clean on the copy fixes that).
  • The sish tunnel keeps StrictHostKeyChecking=no with UserKnownHostsFile=/dev/null: sish host keys may change.
  • serve-web listens on 127.0.0.1 only; nobody uses :8000 on the tailnet IP.
  • A code-tunnel login that's waiting is shown by dev status and docker logs, with no more than that.
  • dev keeps the dev tmux session (windows logs and shell) for people who log in. In browser-editor mode the logs window follows the unit journals; the container's own output is a docker logs codespace hint in dev status.
  • Free disk space in dev status is information only; it doesn't make the codespace unhealthy.
  • OpenTofu mirror (README "OpenTofu mirror"): no fallback to upstream; while the mirror is down tofu init fails. Deleting ~/.tofurc is the only off switch; there's no opt-out variable and no failure hint. Only new home volumes get ~/.tofurc. The tofu wrapper has no test suite.
  • Two browser shells (ttyd), both plain login shells with nothing run on connect: the codespace shell inside the container and the VM shell on the host. Work outside tmux ends when the page disconnects. Login is the tunnel's Google SSO plus basic auth as vscode with one shell_token for both, separate from the editor's token; there's no owner check, same as the editor. Every codespace gets them, with or without the browser editor. No prompt tag or login banner tells them apart; the tab titles and the Slack labels do.
  • The codespace shell is a docker exec per connection, with no -u, -w or --env-file: it gets the container's user and docker group, directory and environment, like the editor's terminal. A disconnected docker exec leaves its shell running in the container, so codespace-ttyd --connect tags each connection (CODESPACE_SHELL_ID) and, when ttyd hangs up, hangs up that shell's session in the container as root, as a real hangup would: tmux, screen and setsid'd jobs (their own sessions) and nohup'd ones keep running.
  • The editor and each shell have their own sish tunnel (one template, one script), so a refused name on one doesn't drop the others. systemctl disable --now codespace-tunnel@<id> is the off switch.
  • A shell fault is information only in dev status; it doesn't make the codespace unhealthy.
  • Accepted edge cases, not fixed:
    • A tmux server started in the VM shell outlives ttyd (KillMode=process), so systemd may report left-over processes in the unit.
    • After the shell token changes, ttyd keeps the old password until it restarts.
    • A codespace shell whose wrapper is SIGKILLed (not hung up), or whose page closes while dockerd is down, leaves its shell in the container until the container restarts.
    • Typing exit in the codespace shell while dockerd is too slow to answer shows the waiting screen instead of closing the page.
    • dev reset doesn't wait for an ensure run already in progress.
    • dev reset with no terminal just refuses to run; there's no --yes flag.
    • A missing .pub beside an existing tunnel key isn't repaired.
    • A failed cde-boot is retried only when the container is next created or started, not after Docker's own restarts.
    • A container in restarting is treated as running.
    • Nothing guards against a broken heredoc end marker beyond the extraction check.
    • The tmux session name isn't protected against a session called e.g. devbox.
    • Two dev runs creating the session at the same moment isn't handled.