feat(tests): runnable docker setup for the preview Maestro SDK skills

[bai-uipath]

What

Makes a docker-driver run of the three Flow v2 (Maestro builder-SDK) preview skills reproducible, and unifies uip auth across the docker experiments.

• tests/experiments/flow-v2-preview.yaml: preview/ as the only skill catalog, a pre_run that writes the @uipath npmrc into $HOME, and the repo-root mount restored
• login state mounts at /.uipath in the preview experiment and in same-ground-headtohead.yaml, matching nightly.yaml
• tests/.coder-eval-version: 0.11.1 -> 0.11.2, the release that carries the mount-destination fix the repo-root mount needs (see The framework pin)
• .github/workflows/smoke-rpa-skills.yml: excludes one broken upstream claude-agent-sdk release on the Windows runner (see The Windows RPA smoke)

Why

Each piece was derived the hard way during an ad-hoc run of all 243 Maestro tasks. They share a failure mode: a missing one scores as a capability failure rather than a configuration error, so the run completes and reports a plausible number that isn't real. In the run that produced this PR, 148 of 257 rows were told the SDK was not installed when they tried to compile.

npm auth. The image ships no runtime npmrc: the build-time one carries a literal token and is deleted in the same layer. Without a replacement, in-sandbox npm install @uipath/flow-sdk 404s against the public registry and every builder-SDK compile fails with it. Written from pre_run rather than baked into tests/docker/Dockerfile, so npm resolution is unchanged for every other suite: several skills tell agents to npm install -g @uipath/cli, which is public-npm-only for stable releases. pre_run runs inside the sandbox before the agent and fails the task loudly, so a broken write cannot masquerade as a capability miss. ${NODE_AUTH_TOKEN} stays literal; npm expands it at read time.

The repo-root mount. plugins.path is both the skill catalog and an automatic bind mount. Narrowing the catalog to preview/ narrows the mount, and criteria across the flow and case suites shell out to $SKILLS_REPO_PATH/tests/tasks/**/_shared/*.py, which then exit 2 and score as ordinary misses. Nightly never sees this because its plugins.path is the repo root.

Auth. Standardizing on nightly's mount rather than deriving a "more correct" destination: it is the arrangement with a 500-task/night track record and it needs no framework change. This also drops the two /home/tmatup/... hardcodes from same-ground-headtohead.yaml, which keeps its name — per tests/README.md it is a general campaign comparison arm, not the v1 half of a Flow v2 head-to-head.

The framework pin

The repo-root mount's destination has to be the host path, because $SKILLS_REPO_PATH is forwarded with its host value and criteria interpolate it in-container. _validate_extra_mount rejects a non-literal destination, so on 0.11.1 the experiment aborts on load. UiPath/coder_eval#128 shipped in 0.11.2, so tests/.coder-eval-version moves 0.11.1 -> 0.11.2 here.

That is a single patch. #2707 already put main on 0.11.1, so the 0.11.x surface that would otherwise need review here — directory-only task.reference (0.11.0, UiPath/coder_eval#106), the cap-drop anti-cheat window (0.11.0), and the anthropic 1.0.0 / httpx2 fix for llm_judge (0.11.1, UiPath/coder_eval#133) — is already on main and is not introduced by this PR. The only behaviour this bump adds is #128.

One note on the cap-drop rider even though main already carries it, since it is the risk this PR's own verification had to close: on a native-Linux runner whose ~/.uipath is mode 700 owned by a non-root uid, container root could no longer read the login state through the file mount. The ADO dry run below settles it.

The Windows RPA smoke

Green as of f51d19a81. The check only runs when the test infra changes, so this PR is the one that trips it. Two breaks were stacked and neither originates here.

The llm_judge import. anthropic 1.0.0 moved to httpx2 and coder-eval 0.10.2 imported httpx without declaring it, so any task carrying an llm_judge errored at setup. Fixed in 0.11.1, which main already carries.

The Windows agent_crash. claude-agent-sdk is unpinned, and 0.2.144 turns out to be a partial publish: no win_amd64 wheel, no git tag, no GitHub Release, with upstream's releases page still showing 0.2.143 as latest. PyPI's latest is a version that was never released, and it carries no bundled claude.exe, so all three tasks retried 3x on agent_crash - Claude Code not found and scored 0.000. Filed upstream as anthropics/claude-agent-sdk-python#1234.

Waiting on an upstream backfill would block this PR, so the RPA smoke install excludes that single release (claude-agent-sdk!=0.2.144). It resolves to 0.2.143 today and goes inert once upstream backfills the wheel or yanks the version. Verified on this branch: claude-agent-sdk==0.2.143, 3/3 tasks >= 0.7.

Verification

Docker driver on colima, against the released wheel rather than the #128 branch, after merging the post-#2707 main:

The fix itself, at the unit level. _validate_extra_mount called directly on both released wheels from PyPI:

coder-eval 0.11.1   $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro
                    -> ValueError: destination must be an absolute path.
coder-eval 0.11.2   $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro
                    -> /Users/bai.li/uipath/skills:/Users/bai.li/uipath/skills:ro

End to end. Same task, codex on gpt-5.6-luna, host wheel and in-container engine both 0.11.2: SUCCESS, 6/6 criteria at 1.000, 218.3s.

1.00  command_executed  Agent invoked `uip is connections list` to discover available connections
1.00  run_command       Tenant has at least one Jira connection (precondition for `node configure`)
1.00  run_command       Flow file exists and is valid JSON
1.00  run_command       Flow uses Jira connector or task-specific managed HTTP fallback
1.00  run_command       Flow path parameters carry the issue key ENGCE-00000
1.00  command_executed  uip maestro flow validate was called

The repo-root mount reaches the real argv, and 0.11.0's anti-cheat rider is active in the same run:

docker run ... --cap-drop DAC_OVERRIDE --cap-drop DAC_READ_SEARCH ...
  -v /Users/bai.li/uipath/skills:/Users/bai.li/uipath/skills:ro
  -v /Users/bai.li/.uipath:/.uipath:rw
... fs_permissions: set_permissions: /work/task_dir -> 0o0 (depth 1)

The local run cannot settle the cap-drop question on its own: on this host ~/.uipath is drwxr-xr-x with .auth at 644, and colima's virtiofs remaps ownership to root in-container, so the mode-700/non-root-uid case cannot be reproduced.

ADO dry run (codex / gpt-5.6-luna, Linux + Windows). Cap-drop does not break the auth mount on the BYOS agent: HOME is /root, the mount is -v /root/.uipath:/.uipath:rw, the host directory is root-owned and the container runs as root, so dropping DAC_OVERRIDE/DAC_READ_SEARCH is inert there — zero permission errors, and the auth-dependent criteria scored 1.000. The split also validated end to end: the Windows slice succeeded (2/2 on its dry-run cap), and Merge slices + publish one run and Delete BYOS agents both succeeded.

The Linux slice still reported GATE: FAIL, from causes outside this PR: Bedrock returned 403 for account 074103337776 under an Organizations SCP deny (1 task error plus the non-gating analysis and review steps), @dev tool installs reported "No compatible version" for orchestrator-, integrationservice- and maestro-tool (1 task error), one genuine task miss, and the antigravity harness smoke looked for uipath-admin/audit_events_basic_smoke.yaml where the task actually lives at uipath-admin/audit/audit_events_basic_smoke.yaml — a stale path in coder_eval_uipath, present on main too.

[github-actions]

Claude finished @bai-uipath's task in 2m 13s —— View job

---

PR Review: Docker setup for preview Maestro SDK skills

• Read review criteria and project conventions
• Read full diff (origin/main...HEAD)
• Read each changed file in full context
• Check for secrets, hardcoded paths, cross-skill deps
• Check CODEOWNERS, relative links, and conventions
• Post review findings

---

Summary

Moves npm auth for the @uipath scope out of the shared Docker image into a pre_run hook scoped to the new flow-v2-preview.yaml experiment, renames the experiment from preview-maestro-sdk.yaml, removes the 82-line preflight.sh script, and unifies auth mount destinations across same-ground-headtohead.yaml and nightly.yaml to /.uipath. Net result: −190 / +94 lines across 6 files.

Change-by-Change Review

1. tests/docker/Dockerfile (−25 lines)

Severity: OK

Removes the runtime npmrc layer (RUN printf ... > /root/.npmrc + ENV NPM_CONFIG_USERCONFIG) that was the only @uipath-scope auth in the shared image. This is correct: the auth is now written per-experiment via pre_run, which limits the blast radius to suites that actually need GitHub Packages access. The 25 lines of explanatory comments justifying NPM_CONFIG_USERCONFIG are no longer needed and cleanly removed.

2. tests/docker/preflight.sh (deleted, −82 lines)

Severity: OK

The preflight script validated 11 preconditions before a full run. Its functionality is replaced by a simpler one-liner documented in tests/README.md (docker run ... uip login status). Given the PR body's reasoning that an inline login check suffices — the other conditions (npm auth, repo mount) now fail loudly via pre_run exit code and $SKILLS_REPO_PATH resolution respectively — this deletion is justified. No remaining references to preflight.sh exist in the repo.

3. tests/experiments/flow-v2-preview.yaml (new, +75 lines)

Severity: OK

Replaces preview-maestro-sdk.yaml with a cleaner version. Key changes:

• pre_run npm auth (lines 52–67): Correctly writes ${NODE_AUTH_TOKEN} literally via single-quoted printf arguments in the YAML |- block — npm expands it at read time. No token touches disk. The mkdir -p "$HOME" guard handles the case where the forwarded $HOME doesn't exist in-container.
• extra_mounts (lines 34–41): ~/.uipath:/.uipath:rw matches nightly.yaml:31 exactly. The repo-root mount $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro is correctly documented as blocked on coder_eval#128.
• env_passthrough_extra (lines 20–33): Superset of nightly's list plus NODE_AUTH_TOKEN and UIPATH_CLI_DISABLE_VERSION_SYNC. The added env vars are documented with comments explaining parity with nightly.

No issues found. Comments are proportionate and explain the "why" for non-obvious choices (e.g., line 35: "do not 'fix' it").

4. tests/experiments/preview-maestro-sdk.yaml (deleted, −79 lines)

Severity: OK

Superseded by flow-v2-preview.yaml. No remaining references to preview-maestro-sdk anywhere in the repo.

5. tests/experiments/same-ground-headtohead.yaml (+4/−4 lines)

Severity: OK

Replaces two hardcoded /home/tmatup/.uipath mount destinations with /.uipath, matching nightly.yaml. The comment is updated to explain why. The deeper .skills mount at /.uipath/.skills:ro correctly shadows the parent /.uipath:rw mount. No remaining /home/tmatup references exist in the repo.

6. tests/README.md (+15 lines)

Severity: OK

Adds the flow-v2-preview.yaml row to the experiment table and a documentation paragraph explaining the experiment's purpose, the manual pre-run check, and the auth mount. The docker run verification command matches the mount config in the experiment file. Table formatting is consistent with existing rows.

What's Missing

• Minor: flow-v2-preview.yaml omits system_prompt and agent.type, while both nightly.yaml and same-ground-headtohead.yaml include a system_prompt restricting sibling runs/ access. This may be intentional (preview experiments may not need the constraint), but worth confirming the agent won't read sibling run artifacts during campaign runs.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill files changed
E2E Tests	N/A	Test infra only, no skill or task changes
Skill Body	N/A	No skill files changed
References & Assets	OK	README documentation is accurate and consistent
Repo Hygiene	OK	No secrets, no personal paths, no stale references to deleted files

Issues for Manual Review

• The $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro mount destination uses a variable, which the PR documents as blocked on coder_eval#128. Until that ships, running this experiment on the pinned 0.10.2 will abort on load. This is acknowledged in the PR body — just confirming the merge is intentional as prep work.
• The missing system_prompt in flow-v2-preview.yaml should be confirmed as intentional, especially if this experiment runs in campaign mode where sibling run directories exist.

Conclusion

Clean PR. The architecture decision to move npm auth from the shared image layer into a per-experiment pre_run is well-reasoned — it limits blast radius, keeps the token out of the image, and fails loudly rather than silently. The hardcoded path cleanup in same-ground-headtohead.yaml and the deletion of the now-redundant preflight.sh are both correct. No blocking issues. Approve.

---