The GlueOps codespace: the devcontainer image (ghcr.io/glueops/codespaces) with the tools to run the GlueOps
platform, and the VM setup that runs it as a cloud development environment (CDE) created from Slack.
.devcontainer/Dockerfile: the codespace image..devcontainer/tools/holds the tools installed into it (cde-boot,cde-init,cde-trust-serve-web,captain_utils,gluekube_ssh, ...), including thetofuwrapper (tools/tofu.sh; the real binary is/usr/local/libexec/tofu)..devcontainer/misc-configs/tofurcbecomes~/.tofurc(README "OpenTofu mirror").developer-setup.sh: VM setup, run at golden-image build (Packer) and by the break-glasscurl setup.glueops.dev | bash. It embeds the VM's scripts and systemd units as heredocs, so it must stay a single self-contained file:codespace-ensure,codespace-tunnel,codespace-ttyd,dev,/usr/local/lib/codespace/common.sh, and thecodespace-ensure.{service,timer},codespace-tunnel@.service(instanceseditor,shell,vm) andcodespace-ttyd@.service(instancesshell,vm) units. README "The codespace VM" explains them, and its "Browser shells" section has the files the Slack bot writes.qemu.pkr.hcl,os-setup-start.sh,os-setup-finish.sh: the Packer VM image.hack/: test runners (hack/test.shruns them all) andhack/vm-recovery-test.sh(manual, against a real VM).tests/vm/: behaviour tests and stubs for the VM scripts. App tests live under.devcontainer/tests/<app>/.
hack/test-codespace-vm.sh: the VM scripts and units, inside a pinned debian:13 container (needs docker).hack/test-captain-utils.sh,hack/test-gluekube-ssh.sh: the image's tools.hack/test-app-chart-preflight.sh:app-chart-preflightagainst stubbedkubectlandargocd, no cluster. Runs on the host and needs bash, jq, yq (mikefarah v4), python3 (the stub Argo CD server), curl, diff and timeout.hack/test.sh: all of the above (CODESPACE_VM_TEST_DOCKER=1to include the VM suite).- Each suite has its own workflow under
.github/workflows/with path triggers.
- Each embedded file in
developer-setup.shhas its own quoted heredoc delimiter (<<'CODESPACE_..._EOF'), never<<-, and the test runner extracts files by those delimiters. Nothing it runs at setup time may read stdin (curl | bashfeeds the script on stdin). - Units that run after cloud-final must be
WantedBy=cloud-init.target, nevermulti-user.target: cloud-final is ordered after multi-user.target, and the cycle makes systemd drop the job at boot. The VM test suite checks this.codespace-ttyd@<id>is wanted bycodespace-tunnel@<id>instead, which is wanted by cloud-init.target. - The tokens (
/etc/glueops/cde_token,/etc/glueops/shell_token) never go on a command line, into the environment, into logs or into a URL the VM prints.code serve-webprints the editor's on stdout, so its stdout is discarded. One exception: the shell token on ttyd's-c(ttyd has no credential-file option), sopsandsystemctl status codespace-ttyd@<id>show it; hints point tojournalctl, neversystemctl status. journald stores the command line with every line a process logs, so ttyd's output is discarded too (the script's own lines, before it execs ttyd, still reach the journal). Leak checks look for each token and for the base64 ofvscode:<shell token>, which is what ttyd prints. - Names: each thing the VM serves through sish has a short id (
editor,shell,vm, listed inCODESPACE_ACCESSin common.sh) used unchanged in machine names: units (codespace-tunnel@<id>,codespace-ttyd@<id>), script arguments, subdomains (<id>-<hostname>; the editor keeps the bare hostname) and the common.sh helpers (codespace_bind,codespace_url,codespace_port,codespace_token_file). People see its label: "Editor", "Codespace shell", "VM shell". In those names and labels, "codespace" means the container, "VM" the host, and "terminal" only VS Code's own; a bare "tunnel" is a sish tunnel, the editor's other mode is "VS Code tunnel". Something new gets an id and a label and follows the same names. packer-qemu.ymlis triggered byworkflow_run, so it always runs the default branch's copy: a tag builds its ownqemu.pkr.hclwithmain's workflow (runner, env,PKR_VAR_*). A pre-release can't test edits to that workflow, andqemu.pkr.hcl's defaults must work undermain's.- Conventional Commits, release-please friendly.
Settled; reviewers shouldn't re-raise these.
- VM and boot changes may be breaking. Existing VMs are not migrated: people create a new codespace from Slack.
- No memory cap on the container and no swap file. The OOM kills come from other workloads; recovery handles them.
- The editor runs as the container's main process, supervised by Docker's restart policy plus the per-minute
codespace-ensure. A crash of that process also ends shells and jobs inside the container. That's accepted. - No automatic recreate of a container that won't start;
dev statuspoints todev reset. - The home and
/nixvolumes keep the home contents from the image tag they were first created with, even if a newer tag later runs on the same VM. Upgrading means a new VM. Volumes are not a backup. - The browser editor's VS Code server is pinned (
serve-web --commit-id) to the one baked into the image: a VM gets no VS Code updates in place, including security releases, until a new VM. In exchange there's no ~700 MB download on a browser connect and the workspace-trust shim always applies. - VS Code is one version,
ARG VERSION_VSCODEin the Dockerfile (code package, CLI, and the baked server via the CLI's commit). New images get a newer VS Code only when Renovate bumps it;renovate.jsonexempts it from the org's 30-day wait so security releases reach new images quickly. - The browser editor runs with
--disable-telemetry.code tunnelhas no such flag; its users' own VS Code settings apply. - Credentials kept in the home volume survive a container recreate.
dev resetremoves them. - cloud-init uses
manual_cache_clean: true(os-setup-finish.sh): it discovers its datasource on first boot only, so a stop/start on libvirt/Proxmox (seed attached only at first boot) doesn't re-probe for ~6 min or regenerate host keys. Accepted: a VM copied from an already-booted disk isn't re-initialised (cloud-init cleanon the copy fixes that). - The sish tunnel keeps
StrictHostKeyChecking=nowithUserKnownHostsFile=/dev/null: sish host keys may change. - serve-web listens on 127.0.0.1 only; nobody uses :8000 on the tailnet IP.
- A code-tunnel login that's waiting is shown by
dev statusanddocker logs, with no more than that. devkeeps thedevtmux session (windowslogsandshell) for people who log in. In browser-editor mode thelogswindow follows the unit journals; the container's own output is adocker logs codespacehint indev status.- Free disk space in
dev statusis information only; it doesn't make the codespace unhealthy. - OpenTofu mirror (README "OpenTofu mirror"): no fallback to upstream; while the mirror is down
tofu initfails. Deleting~/.tofurcis the only off switch; there's no opt-out variable and no failure hint. Only new home volumes get~/.tofurc. Thetofuwrapper has no test suite. - Two browser shells (ttyd), both plain login shells with nothing run on connect: the codespace shell inside the
container and the VM shell on the host. Work outside tmux ends when the page disconnects. Login is the tunnel's
Google SSO plus basic auth as
vscodewith oneshell_tokenfor both, separate from the editor's token; there's no owner check, same as the editor. Every codespace gets them, with or without the browser editor. No prompt tag or login banner tells them apart; the tab titles and the Slack labels do. - The codespace shell is a
docker execper connection, with no-u,-wor--env-file: it gets the container's user and docker group, directory and environment, like the editor's terminal. A disconnecteddocker execleaves its shell running in the container, socodespace-ttyd --connecttags each connection (CODESPACE_SHELL_ID) and, when ttyd hangs up, hangs up that shell's session in the container as root, as a real hangup would: tmux, screen and setsid'd jobs (their own sessions) and nohup'd ones keep running. - The editor and each shell have their own sish tunnel (one template, one script), so a refused name on one doesn't
drop the others.
systemctl disable --now codespace-tunnel@<id>is the off switch. - A shell fault is information only in
dev status; it doesn't make the codespace unhealthy. - Accepted edge cases, not fixed:
- A tmux server started in the VM shell outlives ttyd (
KillMode=process), so systemd may report left-over processes in the unit. - After the shell token changes, ttyd keeps the old password until it restarts.
- A codespace shell whose wrapper is SIGKILLed (not hung up), or whose page closes while dockerd is down, leaves its shell in the container until the container restarts.
- Typing
exitin the codespace shell while dockerd is too slow to answer shows the waiting screen instead of closing the page. dev resetdoesn't wait for an ensure run already in progress.dev resetwith no terminal just refuses to run; there's no--yesflag.- A missing
.pubbeside an existing tunnel key isn't repaired. - A failed
cde-bootis retried only when the container is next created or started, not after Docker's own restarts. - A container in
restartingis treated as running. - Nothing guards against a broken heredoc end marker beyond the extraction check.
- The tmux session name isn't protected against a session called e.g.
devbox. - Two
devruns creating the session at the same moment isn't handled.
- A tmux server started in the VM shell outlives ttyd (