Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ LOOPWORKS_ALLOWED_GITHUB_ORGS=""
LOOPWORKS_AGENT_READY_LOOP_ENABLED="true"
LOOPWORKS_DEVELOPMENT_LOOP_ENABLED="true"
LOOPWORKS_RESEARCH_LOOP_ENABLED="true"
LOOPWORKS_PORTAL_DATA_MODE=""
LOG_LEVEL="info"
DATABASE_URL="postgres://loopworks:loopworks@127.0.0.1:5432/loopworks"
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
Expand Down
36 changes: 36 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,39 @@ jobs:
run: bun run test:e2e
env:
LOOPWORKS_AUTH_BYPASS: "true"

seeded-postgres-e2e:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:17-alpine
env:
POSTGRES_DB: loopworks_e2e
POSTGRES_PASSWORD: loopworks
POSTGRES_USER: loopworks
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U loopworks -d loopworks_e2e"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- name: Checkout
uses: actions/checkout@v5

- name: Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: 1.3.14

- name: Install dependencies
run: bun install --frozen-lockfile

- name: Install Playwright browsers
run: bunx playwright install --with-deps chromium

- name: Seeded Postgres Playwright
run: bun run test:e2e:seeded
env:
DATABASE_URL: postgres://loopworks:loopworks@127.0.0.1:5432/loopworks_e2e
35 changes: 33 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Copy `.env.example` to `.env.local` for local development. The fixture server on
- `LOOPWORKS_AGENT_READY_LOOP_ENABLED`
- `LOOPWORKS_DEVELOPMENT_LOOP_ENABLED`
- `LOOPWORKS_RESEARCH_LOOP_ENABLED`
- `LOOPWORKS_PORTAL_DATA_MODE`
- `LOG_LEVEL`
- `DATABASE_URL`
- `OTEL_EXPORTER_OTLP_PROTOCOL`
Expand Down Expand Up @@ -87,6 +88,27 @@ bun run storybook:build
bun run test:e2e
```

`bun run test:e2e` owns a fresh development server with explicit non-production
fixture mode. It deterministically verifies the `Fixture fallback` path and
does not attach to an existing server. `LOOPWORKS_PORTAL_DATA_MODE=fixtures`
is ignored in production, where database failures continue to fail closed.

The seeded Postgres browser lane is separate. It requires a local
`loopworks_e2e` database owned by the `loopworks` role:

```bash
createdb --host 127.0.0.1 --username loopworks loopworks_e2e
DATABASE_URL="postgres://loopworks:loopworks@127.0.0.1:5432/loopworks_e2e" \
bun run test:e2e:seeded
```

The seeded command refuses production runtimes, non-Postgres URLs,
non-loopback hosts, and database names other than `loopworks_e2e` before it
runs migrations. It then runs migrations, resets only the fixed-id demo rows,
and requires `Live database` browser evidence. Migration or seed failures name
the failed stage; confirm Postgres is running and that the local role and
database exist.

The aggregate command is:

```bash
Expand Down Expand Up @@ -120,7 +142,10 @@ The pre-commit hook runs `bun run precommit`, which mirrors CI validation: Biome

## Database Seed Data

After the database bootstrap (`bun run db:migrate`), seed a demo dataset covering repos, loops, runs, run steps, artifacts, approvals, and Vercel deployment states in every status:
After the database bootstrap (`bun run db:migrate`), seed a demo dataset
covering repos, loops, runs, run steps, artifacts, approvals, and Vercel
deployment states in every status. `DATABASE_URL` must explicitly identify a
local Postgres database:

```bash
bun run db:seed
Expand All @@ -134,4 +159,10 @@ bun run db:seed:reset

Reset only deletes the exact rows this script owns, not the whole table, so any other data in those tables is left untouched.

Add `-- --dry-run` to either command to print the planned row counts without writing. Per [ADR 0007](docs/adr/0007-explicit-seed-data-and-fixture-policy.md), both commands refuse to run when `NODE_ENV` or `VERCEL_ENV` is `production`, or when `DATABASE_URL` does not point at a loopback host (`localhost`/`127.0.0.1`/`::1`) — Loopworks demo data must never write into a database that isn't obviously local.
Add `-- --dry-run` to either command to print the planned row counts without
writing. Per
[ADR 0007](docs/adr/0007-explicit-seed-data-and-fixture-policy.md), both
commands refuse to run when `DATABASE_URL` is missing or malformed, when
`NODE_ENV` or `VERCEL_ENV` is `production`, or when the URL is not Postgres on
a loopback host (`localhost`/`127.0.0.1`/`::1`) — Loopworks demo data must never
write into a database that isn't explicitly and obviously local.
22 changes: 17 additions & 5 deletions docs/adr/0007-explicit-seed-data-and-fixture-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,19 @@ Seed data must never contain real tokens, private keys, customer data, or secret
3. Logs include fallback reasons without logging secrets or raw payloads.
4. Playwright and Storybook exercise fixture data intentionally.
5. Production environments reject unsafe local auth bypasses and unsupported in-memory stores.
6. `bun run db:seed` / `bun run db:seed:reset` refuse to run when either the
runtime looks like production (`NODE_ENV`/`VERCEL_ENV`) or `DATABASE_URL`
does not point at a loopback host, before touching the database. Portal
6. `bun run db:seed` / `bun run db:seed:reset` require an explicit Postgres
`DATABASE_URL` and refuse to run when either the runtime looks like
production (`NODE_ENV`/`VERCEL_ENV`) or the URL does not point at a loopback
host, before touching the database. Portal
pages that are still fixture-only render an explicit degraded notice, and
log a structured warning identifying the fixture-gate reason, instead of
fixture data in production. See `tests/unit/scripts/seed-demo-data.test.ts`,
`tests/unit/lib/runtime.test.ts`, and `tests/unit/portal/fixture-gated-page.test.tsx`.
7. Database-mutating test orchestration validates its target before migrations
as well as before seeding. The seeded Playwright lane requires the dedicated
loopback `loopworks_e2e` database; the normal Playwright lane enables an
explicit non-production fixture mode and proves fixture fallback. Production
ignores that mode and continues to fail closed.

## Follow-Ups

Expand All @@ -50,8 +56,8 @@ Seed data must never contain real tokens, private keys, customer data, or secret
Verified by `tests/unit/seed/demo-data.test.ts`.
3. **Done.** Tests that fixture fallback cannot run in production:
`scripts/seed-demo-data.ts` checks `isProductionRuntime()` **and** that
`DATABASE_URL` resolves to a loopback host, before any database dependency
is touched, and refuses to seed if either check fails
`DATABASE_URL` is explicit Postgres and resolves to a loopback host, before
any database dependency is touched, and refuses to seed if either check fails
(`tests/unit/scripts/seed-demo-data.test.ts`). The `DATABASE_URL` check
exists because a runtime-label check alone is not sufficient: an
operator's local shell can have `DATABASE_URL` pointed at a real Postgres
Expand Down Expand Up @@ -80,3 +86,9 @@ Seed data must never contain real tokens, private keys, customer data, or secret
`src/components/ui/status-badge.tsx` if it introduces a new status value.
- Add or update the relevant Storybook story so the state is reviewable in
isolation, per `src/components/AGENTS.md`.
5. **Done.** Seeded Postgres Playwright coverage uses a separate command and CI
job. Its preparation runner validates the dedicated `loopworks_e2e` target
before migrations, then migrates, resets the fixed-id demo data, and runs
browser assertions that require live database labels and representative
rows across dashboard, catalog, loops, approvals, and settings. The fixture
Playwright command remains separate and deterministic.
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"test": "vitest run",
"test:watch": "vitest",
"test:e2e": "playwright test",
"test:e2e:seeded": "bun run scripts/test-seeded-postgres.ts",
"storybook": "storybook dev -p 6006",
"storybook:build": "storybook build",
"review:ui": "bun run scripts/review-ui.mjs",
Expand Down
5 changes: 4 additions & 1 deletion playwright.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,11 @@ export default defineConfig({
webServer: {
command: "bun run dev:fixture",
url: "http://127.0.0.1:3000",
reuseExistingServer: !process.env.CI,
reuseExistingServer: false,
timeout: 120_000,
env: {
LOOPWORKS_PORTAL_DATA_MODE: "fixtures",
},
},
projects: [
{
Expand Down
31 changes: 31 additions & 0 deletions playwright.seeded.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
testDir: "./tests/e2e",
fullyParallel: true,
forbidOnly: Boolean(process.env.CI),
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: process.env.CI ? "github" : "list",
use: {
baseURL: "http://127.0.0.1:3100",
trace: "on-first-retry",
},
webServer: {
command: "bun run dev:fixture -- -p 3100",
url: "http://127.0.0.1:3100",
reuseExistingServer: false,
timeout: 120_000,
env: {
DATABASE_URL: process.env.DATABASE_URL ?? "",
LOOPWORKS_PORTAL_DATA_MODE: "",
},
},
projects: [
{
name: "seeded-postgres",
use: { ...devices["Desktop Chrome"] },
testMatch: /seeded-postgres\.spec\.ts/,
},
],
});
70 changes: 70 additions & 0 deletions scripts/local-database-safety.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import { isIP } from "node:net";

import { isProductionRuntime } from "@/lib/runtime";

export type LocalDatabaseSafetyOptions = {
requiredDatabaseName?: string;
requireExplicitUrl?: boolean;
};

function isLoopbackHostname(hostname: string): boolean {
return (
hostname === "localhost" ||
hostname === "::1" ||
hostname === "[::1]" ||
(isIP(hostname) === 4 && hostname.split(".")[0] === "127")
);
}

/**
* Returns a sanitized reason when a database target is unsafe for local
* mutation. The URL itself is never included because it may contain credentials.
*/
export function getLocalDatabaseSafetyError(
env: Partial<NodeJS.ProcessEnv>,
options: LocalDatabaseSafetyOptions = {},
): string | null {
if (isProductionRuntime(env)) {
return (
"Refusing local database mutation: this looks like a production environment " +
"(NODE_ENV or VERCEL_ENV is 'production'), per ADR 0007."
);
}

const value = env.DATABASE_URL;
if (!value) {
return options.requireExplicitUrl
? "Refusing local database mutation: DATABASE_URL must explicitly identify a local Postgres database."
: null;
}

let parsed: URL;
try {
parsed = new URL(value);
} catch {
return "Refusing local database mutation: DATABASE_URL is not a valid URL.";
}

if (parsed.protocol !== "postgres:" && parsed.protocol !== "postgresql:") {
return "Refusing local database mutation: DATABASE_URL must use the postgres or postgresql scheme.";
}

if (!isLoopbackHostname(parsed.hostname)) {
return (
"Refusing local database mutation: DATABASE_URL must point at a loopback host " +
"(localhost/127.0.0.1/::1), per ADR 0007."
);
}

if (options.requiredDatabaseName) {
const databaseName = parsed.pathname.replace(/^\//, "");
if (databaseName.includes("%")) {
return "Refusing local database mutation: the database name must not use percent-encoding.";
}
if (databaseName !== options.requiredDatabaseName) {
return `Refusing local database mutation: this command requires the dedicated ${options.requiredDatabaseName} database.`;
}
}

return null;
}
47 changes: 4 additions & 43 deletions scripts/seed-demo-data.ts
Original file line number Diff line number Diff line change
@@ -1,39 +1,13 @@
#!/usr/bin/env bun

import { isIP } from "node:net";

import { db } from "@/db/client";
import { isProductionRuntime } from "@/lib/runtime";
import {
buildDemoSeedData,
type SeedCounts,
type SeedDatabase,
seedDemoData,
} from "@/lib/seed/demo-data";

const defaultLocalDatabaseUrl = "postgres://loopworks:loopworks@127.0.0.1:5432/loopworks";

/**
* Mirrors `isLoopbackWebhookUrl` in `scripts/github-webhook-fixture.ts`. The
* production-runtime check alone is not enough: an operator's shell can have
* `DATABASE_URL` pointed at a real (non-production-labeled) Postgres host
* while `NODE_ENV`/`VERCEL_ENV` are unset. Seeding must refuse that too.
*/
function isLoopbackDatabaseUrl(value: string): boolean {
let parsed: URL;
try {
parsed = new URL(value);
} catch {
return false;
}

return (
parsed.hostname === "localhost" ||
parsed.hostname === "::1" ||
parsed.hostname === "[::1]" ||
(isIP(parsed.hostname) === 4 && parsed.hostname.split(".")[0] === "127")
);
}
import { getLocalDatabaseSafetyError } from "./local-database-safety";

export type RunSeedCliDependencies = {
database: SeedDatabase;
Expand Down Expand Up @@ -97,22 +71,9 @@ export async function runSeedCli(
return 1;
}

if (isProductionRuntime(env)) {
console.error(
"Refusing to seed demo data: this looks like a production environment " +
"(NODE_ENV or VERCEL_ENV is 'production'). Loopworks demo seed data must " +
"never run against production, per ADR 0007.",
);
return 1;
}

if (!isLoopbackDatabaseUrl(env.DATABASE_URL ?? defaultLocalDatabaseUrl)) {
console.error(
"Refusing to seed demo data: DATABASE_URL does not point at a loopback " +
"host (localhost/127.0.0.1/::1). Loopworks demo seed data must only run " +
"against a local Postgres instance, per ADR 0007 - a production-labeled " +
"runtime is not the only way this could write into a real database.",
);
const safetyError = getLocalDatabaseSafetyError(env, { requireExplicitUrl: true });
if (safetyError) {
console.error(safetyError);
return 1;
}

Expand Down
Loading
Loading