Skip to content

feat(mcp): add package-first enterprise mock lab#2670

Draft
reachjalil wants to merge 5 commits into
different-ai:devfrom
reachjalil:feature/mcp-diagnostic-mock-server
Draft

feat(mcp): add package-first enterprise mock lab#2670
reachjalil wants to merge 5 commits into
different-ai:devfrom
reachjalil:feature/mcp-diagnostic-mock-server

Conversation

@reachjalil

@reachjalil reachjalil commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Important

Foundation status: automated verification complete; two independent audits found no package blocker. Verified by Jalil: not started. This remains a draft while Jalil reviews it in small, understandable steps.

This PR is now intentionally package-first. It does not modify Den, the existing generic/Gmail mock, or the current MCP UI. The follow-up Den Test connection PR will be stacked on this branch so the reusable test foundation can be accepted first.

Remote checks: all GitHub test/build/i18n/Helm checks passed, including the new enterprise mock safety gate. The three Vercel preview checks report “Authorization required to deploy” for the fork; that is an external preview-access limitation, not a compile or test failure.

Managerial summary

We set out to make enterprise MCP failures reproducible without needing a customer ServiceNow or Microsoft tenant for every development cycle.

The earlier version mixed a mock server, Den connection behavior, UI behavior, and proof code in one change. This revision establishes a cleaner standard:

  1. a reusable TypeScript package owns enterprise OAuth/MCP profiles, contracts, faults, runtime behavior, probes, and safe diagnostic evidence;
  2. a standalone EE development app owns the protected administrator UI and the lifecycle of local mock instances; and
  3. Den integration remains a separate network consumer, to be reviewed only after this foundation is understood and accepted.

The practical result is a deterministic local lab where a developer can select a provider shape, inject one precise fault, and prove the exact first phase and error category observed on the wire.

What existed, what was added, and why it matters

Before this PR Added by this PR Practical outcome
Enterprise mock behavior lived beside older script and Den concerns. @openwork/enterprise-mcp-mock-server in packages/enterprise-mcp-mock-server. The reusable concern has one explicit public package boundary and can support future tests without importing Den.
There was no dedicated environment for creating and controlling multiple enterprise-style fixtures. @openwork-ee/enterprise-mock-lab in ee/apps/enterprise-mock-lab. A developer can create, start, stop, reset, delete, and probe isolated loopback data planes from a protected local UI.
A failed connection could be described too generally. Declarative expectations and observed evidence retain the first failed phase, stable category, diagnostic level, and safe request evidence. We can distinguish where the failure came from instead of showing only “failed to connect.”
Green initialization/catalog behavior could be mistaken for real provider readiness. Three explicit diagnostic levels and honest probe modes. Connection, catalog readiness, and provider-operation behavior are not conflated.
Provider examples could drift or overstate fidelity. Version-pinned provider profiles with per-aspect fidelity labels, known limitations, and primary documentation links. Reviewers can see what is documented, what is synthetic, and when the fixture was last verified.
Manual enterprise OAuth could not be rehearsed safely unless the mock registered Den's exact callback. The lab accepts 1–10 validated, unique exact OAuth redirect URIs and shows the escaped list while stopped. A developer can obtain Den's callback first, configure the ServiceNow/Microsoft-style client exactly, then start and Connect.
Local test success was not automatically protected in CI. A focused Node 24 / pnpm 11.4 / Bun 1.3.10 workflow runs both safety suites with a frozen lockfile. Future changes cannot silently break this foundation while relevant CI remains green.

Package-first structure

packages/enterprise-mcp-mock-server
  contracts/       provider, scenario, fault, OAuth, tool and runtime contracts
  profiles/        dated declarative ServiceNow and Microsoft fixture profiles
  faults/          named, phase-specific fault catalog
  protocol/        OAuth and MCP wire behavior
  runtime/         bounded instance state and lifecycle
  testing/         loopback-only diagnostic probe

ee/apps/enterprise-mock-lab
  control plane    protected administrator lifecycle API
  data planes      separate package-backed OAuth/MCP listeners
  admin UI         scenario configuration and expected-versus-observed evidence

The app imports the package through its public contract. The package does not import Den routes, Den persistence, Den UI, or organization/member concepts.

The existing scripts/mock-oauth-mcp-server.mjs, Den API/Web paths, turbo.json, and existing Den eval helpers remain byte-identical to current upstream/dev in this PR.

Three diagnostic levels

Level What it answers Representative failures
Connection Can the client reach the server and complete OAuth/MCP transport setup? missing challenge, malformed metadata, issuer mismatch, PKCE/DCR/client/grant/resource/scope failure, TCP refusal, MCP version/init/session/content/SSE failure
Readiness Did initialization complete and is the advertised catalog safe and usable? empty catalog, cursor loop, duplicate tool, invalid tool schema
Operation Can a real provider-style operation pass its authorization and execution boundary? provider authorization/ACL denial, policy denial, throttling, unavailability, timeout after commit

An empty but protocol-valid catalog is reported as a readiness failure, not a connection failure.

For example, the browser proof configures a provider authorization denial and independently observes:

Expected: failure · PROVIDER_AUTHORIZATION · provider_authorization_denied
Observed: failure · PROVIDER_AUTHORIZATION · provider_authorization_denied

The healthy fixture-conformance probe completes OAuth, MCP initialization, the exact pinned tool-name set, and general schema-validity checks. It explicitly says that it did not execute a provider tool.

Provider profiles and fidelity

All profiles are pinned to fixture version 2026-07-12.1 and separately label endpoint, authorization, catalog, schema, result, and transport fidelity.

Profile Purpose Important boundary
Synthetic enterprise OAuth/MCP Standards-oriented control fixture Entirely synthetic provider behavior
ServiceNow inbound Quickstart Manual/confidential-client topology, Quickstart path and scope Synthetic tenant, ACL, results, and provider authorization
Microsoft Work IQ Current documented endpoint/tool-input shape Synthetic consent, results, and mock-only approval/idempotency extensions
Microsoft Enterprise MCP Current enterprise endpoint, resource/audience, .default scope and catalog shape Synthetic Entra tenant, consent, schemas/results where documentation is incomplete
Agent 365 Mail dated preview Pinned preview endpoint and tool names Preview contract; schemas/results are intentionally synthetic and dated

Primary references used by the fixtures:

Safety and correctness

  • Control and provider data planes use separate listeners and literal loopback addresses.
  • The admin app requires a 32-character secret, constant-time authentication, bounded login attempts, expiring HttpOnly / SameSite=Strict sessions, exact Origin checks, and CSRF tokens.
  • Mutating bodies are streamed through a 64 KiB limit only after Origin/session checks; secrets are write-only and excluded from HTML, JSON, events, and probe errors.
  • Public profiles, scenarios, catalogs, faults, snapshots, and events are deeply immutable.
  • OAuth models manual/DCR clients, exact redirects, PKCE S256, scopes/resources, access versus refresh lifetime, refresh rotation, client-bound revocation, and stable session lineage.
  • JSON-RPC request, response, notification, media negotiation, pagination, protocol-version, and session behavior are asserted at the wire boundary.
  • Events, sessions, requests, and mutation records are bounded. Unresolved mutations cannot be evicted; capacity fails closed rather than losing reconciliation evidence.
  • Scenario activation is transactional. Failed activation restores the prior revision; deletion cannot race with a lifecycle operation and resurrect an instance.
  • Probes refuse non-loopback targets, enforce deadlines and 1 MiB response bounds, redact hostile echoes, and distinguish TCP reachability from HTTP routing.
  • Safe-read mode refuses an explicit mutation before it is prepared.

What testing found and how serious it was

Finding Severity before correction Current state
Stale or overstated Microsoft endpoints, scopes, schemas, and fidelity High: a mock could teach the wrong integration contract Corrected, dated, source-linked, and split into per-aspect fidelity claims
OAuth PKCE, refresh/session lineage, and client-bound revocation gaps High: tests could pass behavior a real secure client must reject Corrected and covered by wire tests
Mutable public fixtures and transactional lifecycle races High: one test or failed update could contaminate later evidence Deep-frozen and transactionally restored; concurrency tested
A timed-out mutation could lose reconciliation evidence under pressure High: a retry could duplicate a provider action Unresolved outcomes are never evicted; capacity fails closed
Tool/provider errors and TCP refusal could be attributed to the wrong source High for diagnostics: administrators would troubleshoot the wrong owner Exact first phase/category asserted, including closed-port classification
Empty catalog wording could imply connection failure Medium: readiness and reachability would be confused Explicitly classified as protocol-valid readiness failure
The first “invalid tool schema” fixture violated our stricter subset but was still valid JSON Schema 2020-12 High for consumer testing: Den could appear to reject a malformed provider contract that was not actually malformed Wire payload now uses a genuinely invalid required keyword shape, with a direct protocol assertion
The lab UI could not register Den's exact callback for manual/pre-registered OAuth High for practical ServiceNow/Microsoft rehearsal: browser authorization would fail even when the mock was otherwise healthy JSON/form redirect configuration added, bounded to 1–10 unique safe URIs and visible before startup
Control-plane body, session, Origin, secret, and lifecycle boundaries needed hardening High for a configurable fault lab Bounded/authenticated and covered by app/security/integration tests
Fresh upstream rewrite initially lacked lock importers Merge blocker: frozen CI install failed Lock regenerated from current upstream/dev; final diff contains only two new importers
New suites were not selected by existing CI Merge-readiness blocker for a safety foundation Focused CI workflow added

Verification completed

  • Frozen workspace installation: passed with pnpm 11.4.0.
  • Reusable package: 156/156 tests, strict TypeScript check, ESM build, and declaration build.
  • Advertised fault matrix: 115/115 applicable profile/fault combinations observe their real first failing phase.
  • Healthy safe-read matrix: 10/10 profile/transport combinations, with zero mutations.
  • Standalone lab: 19/19 tests, including security, exact callback configuration, and package-backed lifecycle integration, plus production build.
  • Compiled app smoke: /health returned the loopback-only service identity; the private root required authentication; SIGINT closed the listener.
  • Real Chrome Fraimz replay: 6/6 steps — empty lab, running ServiceNow fixture, exact provider-authorization failure, reset preserving the declared fault, healthy recovery, and clean deletion/listener teardown.
  • Independent package/security audit: no P0/P1/P2 blocker found.
  • Independent foundation/diff audit: initial lock and CI gaps found, corrected, and re-audited with no P0-P2 blocker.
  • Existing Den, Den Web, generic mock, turbo.json, and older eval paths: zero diff from current upstream/dev.
  • git diff --check: passed.
  • Remote GitHub Actions: package/lab safety gate, Linux/macOS OpenWork tests, Den API/Web/inference builds, i18n, and Helm all passed. Fork Vercel previews remain authorization-blocked.

The new focused GitHub workflow makes the package and lab checks automatic for relevant changes. The browser story remains an intentional review proof rather than an overclaimed live-provider test.

What this PR does not prove

This is the most important boundary:

  • no real customer ServiceNow tenant was used;
  • no real Microsoft 365, Work IQ, Enterprise MCP, or Agent 365 tenant was used;
  • no customer proxy, corporate CA, outbound allowlist, Conditional Access, consent, licensing, region, or provider patch posture was exercised;
  • the fixture-conformance probe does not invoke a provider tool;
  • synthetic results do not prove provider business behavior or ACLs; and
  • Den does not yet consume this lab in this PR.

Those are not hidden failures of the package. They are the next verification layers and must remain explicit before anyone claims live-provider conformance.

Suggested review path for Jalil

  1. Boundary review: confirm the package/app names and that the package is independent from Den and the old mock.

  2. Start the lab:

    export ENTERPRISE_MOCK_LAB_ADMIN_SECRET="$(openssl rand -base64 32)"
    pnpm dev:enterprise-mock-lab

    Open http://127.0.0.1:8794.

  3. Create the Den OAuth row first: use the planned local lab endpoint and pre-registered client, then copy the exact callback Den displays before pressing Connect.

  4. Create one ServiceNow fixture: paste that callback into the lab's exact redirect list, keep it stopped first, inspect the profile/fidelity statements, then start it and Connect from Den.

  5. Inject one fault: choose provider authorization denied and run the fixture-conformance probe. Confirm the expected and observed phase/category match.

  6. Recover: reset, switch to Healthy baseline, apply the new revision, probe again, then delete the instance.

  7. Record Jalil’s decision here: only after this foundation is understood should the Den Test connection branch be reviewed and merged into it.

Stacking and follow-up plan

This PR remains based on dev and is the package-first foundation.

The next draft PR will use feature/mcp-diagnostic-mock-server as its base and add only the Den administrator Test connection consumer. That review will answer:

  • how Den calls the mock over the real network boundary;
  • how current structured diagnostics show the exact first failed phase/category in the server UI;
  • which connection/readiness checks are safe and read-only; and
  • which provider-operation claims remain unverified.

Live dynamic tracing remains a later layer and should not be merged until the foundation and connection test are separately understood.

Reviewer focus

  • Public contracts and dependency direction in packages/enterprise-mcp-mock-server/src/index.ts and src/contracts/.
  • Provider provenance/fidelity in src/profiles/profiles.ts.
  • First-failure behavior in src/faults/catalog.ts, src/protocol/, and src/testing/probe.ts.
  • Bounded state and mutation reconciliation in src/runtime/.
  • Admin trust boundary in ee/apps/enterprise-mock-lab/src/security.ts, app.ts, and control-plane.ts.
  • Full fault matrix and adversarial tests under both new workspaces.
  • Focused CI selection in .github/workflows/ci-enterprise-mcp-mock.yml.

@vercel

vercel Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 12, 2026 9:03pm

@vercel

vercel Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

@reachjalil is attempting to deploy a commit to the Different AI Team on Vercel.

A member of the Team first needs to authorize it.

@reachjalil

Copy link
Copy Markdown
Collaborator Author

Agent integration rehearsal update — 2026-07-11

This source branch is published at e1569ec and was exercised in the independent combined rehearsal, #2675.

The rehearsal confirmed the realistic pre-registered ServiceNow-style confidential-client path, the read-only catalog test, distinct credential and provider-permission failures, stable UI row targeting, bounded callback replacement, listener cleanup, fail-closed runtime controls, and operation-only denial and throttling faults.

Status remains deliberately separate:

This PR remains based on upstream dev because the controlled parent branch exists only on the Jalil fork. #2674 is the approval ledger; this PR is the source and evidence branch.

@reachjalil reachjalil force-pushed the feature/mcp-diagnostic-mock-server branch from e1569ec to ababe2a Compare July 12, 2026 20:47
@reachjalil reachjalil changed the title feat(mcp): add enterprise diagnostic mock server feat(mcp): add package-first enterprise mock lab Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant