Skip to content

feat(sandbox): pre-warm deno cache from S3 for deco site repos#4145

Open
igoramf wants to merge 34 commits into
mainfrom
feat/deco-site-s3-cache-warmup
Open

feat(sandbox): pre-warm deno cache from S3 for deco site repos#4145
igoramf wants to merge 34 commits into
mainfrom
feat/deco-site-s3-cache-warmup

Conversation

@igoramf

@igoramf igoramf commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Before running deno task dev, the daemon now attempts to download and extract the pre-built Deno module cache from S3 (<owner>/<repo>/cache.tar.zst) — the same path the admin build pipeline produces
  • Non-fatal: if credentials aren't set or the object doesn't exist in S3, logs a message and continues normally (cold first run)
  • Opt-in via the new decoSiteCache.secretName Helm value

Changes

packages/sandbox/daemon/setup/install.ts

  • New trySpawnDecoSiteCache() — shell script that downloads <owner>/<repo>/cache.tar.zst from S3 and extracts into $DENO_DIR; credentials stay as $VAR references in the shell command (never in the command string or process list)

packages/sandbox/daemon/setup/orchestrator.ts

  • stepInstall(): when spawnInstall returns null (deno case), calls trySpawnDecoSiteCache before marking install succeeded

packages/sandbox/image/Dockerfile

  • Added zstd to apt-get install (required to decompress .tar.zst)

deploy/helm/sandbox-env/

  • New optional decoSiteCache block in values.yaml + sandbox-template.yaml to inject DENO_DECO_CACHE_S3_* env vars from a K8s Secret

Activation

Create a K8s Secret with the S3 credentials and set in values.yaml:

decoSiteCache:
  secretName: deco-site-s3-cache
  region: sa-east-1
  bucket: deco-assets-storage

Test plan

  • Deploy sandbox with decoSiteCache.secretName set — verify [deco-site cache] log appears and deno task dev starts faster
  • Deploy without decoSiteCache.secretName — verify sandbox starts normally with no cache-related errors
  • Deploy with invalid S3 credentials — verify [deco-site cache] s3 cache not available is logged and sandbox still starts

🤖 Generated with Claude Code


Summary by cubic

Speeds up the first deno task dev for deco-sites sandboxes by pre-warming the Deno module cache with a 15‑min pre‑signed S3 URL generated by mesh. No AWS credentials or Pod Identity are required in sandbox pods.

  • New Features

    • Mesh reads DECO_CACHE_S3_BUCKET and DECO_CACHE_S3_REGION, assumes awsS3TenantRoleArn via STS (optionally using provisioner keys), and creates a 15‑min pre‑signed URL to <owner>/<repo>/cache.tar.zst using @aws-sdk/s3-request-presigner (AWS default endpoint).
    • Sandbox provider passes owner, name, and denoCachePresignedUrl into TenantConfig; the daemon’s tryWarmDenoCache downloads and restores into $DENO_DIR (non‑fatal on errors).
    • Sandbox image includes zstd for .tar.zst extraction.
  • Migration

    • Configure mesh with DECO_CACHE_S3_BUCKET, DECO_CACHE_S3_REGION, and awsS3TenantRoleArn; remove any sandbox Helm decoCache/DECO_CACHE_S3_* and serviceAccountName values. Remove DECO_CACHE_S3_ENDPOINT if present.
    • Ensure the repo owner and name are provided so mesh can build the cache object path.

Written for commit a930e64. Summary will update on new commits.

Review in cubic

igoramf and others added 30 commits June 25, 2026 14:28
Before running `deno task dev`, the daemon now tries to download and
extract the pre-built Deno module cache from S3 (<owner>/<repo>/cache.tar.zst).
This avoids cold first-run dependency fetching for deco site sandboxes.

- Add `trySpawnDecoSiteCache` to install.ts — non-fatal shell step that
  downloads the cache tarball and extracts it to $DENO_DIR
- Call it in orchestrator.ts `stepInstall()` when spawnInstall returns
  null (deno case), before marking install as succeeded
- Add `zstd` to the sandbox image Dockerfile
- Add optional `decoSiteCache` Helm block in sandbox-env to inject
  DENO_DECO_CACHE_S3_* credentials from a K8s Secret into the pod

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove hardcoded defaults — open-source consumers must configure their
own bucket and region. If either is unset the cache step is skipped.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename trySpawnDecoSiteCache → tryWarmDenoCache (no deco coupling)
- Rename env vars DENO_DECO_CACHE_S3_* → DENO_CACHE_S3_* (generic)
- Add DENO_CACHE_S3_ENDPOINT for any S3-compatible storage (R2, GCS, MinIO)
- Add DENO_CACHE_S3_PATH explicit override; falls back to auto-deriving
  <owner>/<repo>/cache.tar.zst from the GitHub clone URL when unset
- Rename Helm value decoSiteCache → denoCache, add path field

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- DENO_CACHE_S3_PATH_PREFIX: prefix before repo name, e.g. "my-org" →
  "my-org/<repo>/cache.tar.zst"
- DENO_CACHE_S3_PATH_FILE: configurable filename, defaults to
  "cache.tar.zst"
- DENO_CACHE_S3_PATH: full override still supported for complete control
- Daemon now only needs the repo name from the clone URL (not owner)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…-inject path for deco.cx sites

Generic prefix (DECO_) decouples the cache config from the Deno runtime so
it can serve other runtimes in the future without a rename. The mesh now
injects DECO_CACHE_S3_PATH automatically when a Virtual MCP has `siteSlug`
in its metadata (the unique marker for deco.cx imported sites), building the
path as `{owner}/{name}/cache.tar.zst` from the linked GitHub repo — no Helm
pathTemplate needed for deco-sites.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…aemon

The mesh now injects DECO_CACHE_S3_PATH directly from the Virtual MCP
metadata (owner/repo), so the daemon no longer needs to derive it from
the clone URL. Removes resolveCachePathFromTemplate, DECO_CACHE_S3_PATH_TEMPLATE
support, and the pathTemplate Helm value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A static path shared across all sandboxes in an environment doesn't make
sense — DECO_CACHE_S3_PATH is injected per-repo by the mesh.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n decoCache

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fails helm install/upgrade with a clear message when secretName is set
but region or bucket are missing — the daemon would silently skip the
cache warm-up in that case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds optional Mode B alongside the existing "bring your own secret" mode:
when decoCache.externalSecret.enabled=true the chart creates a SecretStore
+ ExternalSecret that pulls credentials from AWS Secrets Manager and
materialises the K8s Secret automatically — no manual kubectl create needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cket

The daemon already handles missing config gracefully — tryWarmDenoCache
returns null and the sandbox starts normally without cache warm-up.
Failing helm install over an optional cache feature is disproportionate.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes the redundant externalSecret.provider.aws.region — decoCache.region
already holds the AWS region and serves both the S3 signing and the
SecretStore config.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
S3 bucket and Secrets Manager can be in different AWS accounts/regions.
Revert the previous assumption that decoCache.region covers both.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of baking DECO_CACHE_S3_* credentials into every sandbox pod via
SandboxTemplate, the mesh now reads them from its own settings
(DECO_CACHE_S3_ACCESS_KEY_ID/SECRET/REGION/BUCKET/ENDPOINT) and injects
them at runtime via PUT /_sandbox/config only for deco.cx sites (identified
by siteSlug in Virtual MCP metadata). Non-deco-site sandboxes never receive
S3 credentials. Removes the decoCache section from sandbox-env chart entirely.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The daemon now auto-detects deco storefronts using two heuristics:
  1. Clone URL owner is deco-sites (primary — covers the official org)
  2. Repo has a .deco/ directory post-clone (secondary — custom orgs)

When detected, it derives the S3 cache path as {owner}/{repo}/cache.tar.zst
and warms the Deno module cache before the first run. DECO_CACHE_S3_PATH
still works as an explicit override. Credentials come from the sandbox pod
env (decoCache in sandbox-env chart) — the mesh is not involved at all.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes heuristic 2 (.deco/ dir) and DECO_CACHE_S3_PATH override.
The daemon now only checks if the clone URL owner is deco-sites.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
No longer needed — deco-site detection and cache path derivation moved
entirely into the daemon.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rsing clone URL

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…owner/name/cache.tar.zst

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…epoName

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…playName format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace curl --aws-sigv4 with @aws-sdk/client-s3. Credentials are now
provided via EKS Pod Identity — no static keys in env vars or secrets.
Remove ExternalSecret template and secretName/accessKeyIdRef fields from
the Helm chart; only region, bucket, and optional endpoint remain.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
igoramf and others added 4 commits June 26, 2026 10:45
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of injecting S3 credentials into sandbox pods (Pod Identity),
mesh now generates a 15-min pre-signed URL scoped to the specific repo's
cache object and passes it to the daemon via TenantConfig.denoCache.
The daemon downloads with fetch() — zero AWS credentials in the pod.

- mesh: add decoCacheS3{Bucket,Region,Endpoint} settings + env var reads
- mesh/start.ts: tryGenerateDecoCachePresignedUrl generates scoped URL
- daemon/install.ts: rewrite tryWarmDenoCache to use presignedUrl via fetch
- sandbox/package.json: remove @aws-sdk/client-s3 (no longer needed)
- sandbox-env chart: remove DECO_CACHE_S3_* env vars, decoCache values,
  and serviceAccountName (sandbox pods need no S3 access at all)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Follow the same pattern as tenant-credentials.ts: assume rl-studio-org-storage
via STS (provisioner key pair if configured, pod identity otherwise) and use
the temporary credentials to sign the S3 URL. Restores DECO_CACHE_S3_REGION
setting needed for the STS client.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants