fix(auth): refresh active-turn credentials per request (ENG-2116)

[lucas-koontz]

User story

As a signed-in Cowork user
I want an active turn to adopt refreshed MindsHub credentials and recover from one isolated authentication refusal
So that long-running work is not discarded or blamed on my account

Hotfix scope

This is #428 replayed onto main so the fix can ship without waiting for the staging backlog. The three ENG-2116 commits are byte-identical to the ones on fix/eng-2116-refresh-active-jwt, confirmed with git range-diff; nothing else from staging came along. #428 stays open against staging, and the backmerge resolves it.

One commit is new here, and it is the only difference from #428: the Anton pin.

The Anton pin

	#428 (staging)	This PR (main)
anton-agent source	rev = "095c17ae" — anton#421's head, cut from Anton staging	branch = "main", lockfile resolving to 18b043f6

The old pin comment claimed Anton main lacks resolve_artifact_id, canonical_artifact_id and artifact_key. That is no longer true: Anton main defines all three in anton/core/artifacts/models.py and exports them. The pin now exists only to reach the ENG-2116 supplier, not to hold the tree on a staging commit. The comment says that.

Resolved. anton#429 merged as 18b043f6, so Anton main now carries the ENG-2116 supplier alongside the artifact-identity symbols. Both reasons for a rev pin are gone and this tracks branch = "main" again, which removes the merge-ordering hazard this file has carried since #407.

Why this matters

The sidecar previously built long-lived model providers from one startup credential. Even after the desktop refreshed and handed over a MindsHub JWT, an active turn could keep sending the expired value and surface the gateway refusal as a user-session problem.

Acceptance criteria

• Local Minds providers for resolved router, planning, and coding roles read the current in-memory runtime credential for each request.
• Organization-mode, BYOK, and other static provider credentials retain their existing behavior.
• A confirmed Anton provider refusal maps to the existing typed API error with the actual failing role.
• Health reports whether a resolved required planning or coding role needs the runtime Minds credential; router-only Minds usage and organization mode report false.
• Existing billing, rate-limit, model, and generic connection mappings remain unchanged.
• Anton pin repointed: back to branch = "main", lockfile at 18b043f6.
• A comparable post-deployment window has no auth-typed failures for the affected paid install.

How to test

1. Construct a local Minds provider with credential A, update the runtime store to B, and verify the next provider request uses B without rebuilding the turn.
2. Repeat for resolved planning-only and coding-only Minds configurations; verify static organization-mode and BYOK configurations do not use the runtime supplier.
3. Raise a typed provider refusal from planning, coding, and verifier work and verify the API error preserves the actual failing role and action.
4. Verify the health flag is true for resolved planning or coding Minds roles, false for direct-provider required roles, false for router-only Minds usage, and false in organization mode.
5. Exercise non-auth provider failures and verify their existing classifications are unchanged.

Notes for the reviewer

The runtime store remains the boundary. The desktop already owns credential rotation and handoff; the sidecar passes an async supplier into local Minds-backed Anton providers so live instances read the store at request time. Static org and user-provided credentials are intentionally unchanged.

The health flag is authoritative for the desktop wake gate. It reflects resolved required planning and coding roles only. Optional router usage cannot delay an otherwise direct-provider turn.

Hosted 401s keep the Reconnect card. The worker pods run the minds-anton-scratchpad image, pinned in scratchpad-controller and bumped independently of this PR set, so remote_turn_error matches the legacy 401 copy as well as the typed name. Both Anton imports are lazy with fallbacks for the same version-skew reason.

Verified locally, against main

Run in a worktree cut from origin/main (6095d840) with these commits replayed on top.

Check	Observed result
Full unit suite (make test)	2,953 passed, 1 skipped, 0 failed (50s)
test_build_llm_client.py / test_health_owner.py / test_turn_errors.py	20 / 16 / 130, 166 passed together
The resolved Anton really exposes the capability	OpenAIProvider(api_key_provider=...) and ProviderAuthError both resolve at 18b043f6 (anton main), so the capability gates take the live branch instead of silently degrading
uv lock --check	Clean, 124 packages resolved
tests/integration	9 self-skipped without Redis / COWORK_BASE_URL
Staging-only symbol sweep over the replayed diff	No findings. runtime_credential and its accessors are already on main, and every file the replayed code depends on is byte-identical between main and staging
git range-diff against #428	All three ENG-2116 commits byte-identical; the pin commit is the only addition

Ships with

Deploys: #444

• Anton: fix(auth): refresh provider credentials per request (ENG-2116) anton#429
• cowork desktop: fix(auth): await refreshed credential handoff (ENG-2116) cowork#806
• scratchpad-controller (worker image pin): https://github.com/mindsdb/scratchpad-controller/pull/41

Merge order: Anton #429 (merged) → this PR → cowork desktop #806.

[github-actions]

PR environment is up · pr-cowork-server-444

Console	https://pr-cowork-server-444.dev.mindshub.ai
Auth API	https://auth-pr-cowork-server-444.dev.mindshub.ai
Cowork Web	https://cowork-pr-cowork-server-444.dev.mindshub.ai
Keycloak Admin	https://auth-pr-cowork-server-444.dev.mindshub.ai/auth/admin/
Inference API	https://api-pr-cowork-server-444.dev.mindshub.ai/v1
Cowork Server API	https://cowork-pr-cowork-server-444.dev.mindshub.ai/api
Health	https://cowork-pr-cowork-server-444.dev.mindshub.ai/api/v1/health/
Namespace	pr-cowork-server-444

Every service is in this namespace. The ones this PR does not build run the staging image unless you link them with Deploys: in the PR body.

_{Updated on every push to this PR.}

[Copilot]

🟡 Changes recommended

It still locks/pins anton-agent to an unmerged branch head (non-reproducible main build) and the lockfile now makes portalocker depend on Windows-only pywin32 unconditionally, which can break non-Windows installs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates Cowork’s Anton-backed provider wiring so active turns can pick up refreshed MindsHub credentials per request, and so confirmed provider-auth failures are classified/messaged accurately (including by role) without regressing other gateway error mappings.

Changes:

• Add a per-request runtime credential supplier for local MindsHub-backed providers, capability-gated for older Anton versions.
• Switch auth failure detection/mapping to Anton’s typed ProviderAuthError, while keeping a narrow legacy fallback for remote worker pods.
• Extend health reporting and tests to reflect when the desktop runtime Minds credential is required.

File summaries

File	Description
cowork/services/providers.py	Adds runtime credential supplier and capability-gated api_key_provider wiring for MindsHub providers.
cowork/handlers/turn_errors.py	Uses typed ProviderAuthError (plus legacy anchored fallback) for auth detection and remote error mapping.
cowork/handlers/responses.py	Attributes auth failures to the actual failing role/provider when possible; avoids incorrect defaults in mixed configs.
cowork/api/v1/endpoints/health.py	Adds minds_runtime_credential_required flag derived from resolved required roles (planning/coding) and org mode.
tests/test_build_llm_client.py	Tests runtime supplier behavior, org/static behavior boundaries, and backward-compat degradation paths.
tests/test_turn_errors.py	Tests typed auth detection, legacy fallback behaviors, and correct reconnectability/provider labeling by role.
tests/test_health_owner.py	Tests health flag behavior for runtime credential requirement across role/provider combinations and org mode.
pyproject.toml	Pins anton-agent to a specific git rev for ENG-2116 capabilities and documents merge-order constraint.
uv.lock	Updates locked Anton rev/deps and reflects resolver output (includes a platform-marker regression to fix).

Review details

• Files reviewed: 8/9 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[alecantu7]

Review — 202fd3b4 (hotfix to main)

I reviewed and approved the staging counterpart (#428 @ a3beb2f3) over four rounds. This carries that content unchanged, so rather than re-derive it I verified equivalence and looked only at what differs.

Verified identical to the approved version

Diffing the ENG-2116 files between a3beb2f3 and this head: responses.py, providers.py, health.py and test_build_llm_client.py are byte-identical. Both findings I raised survive:

• the hosted-401 mapping at :950-954 — typed branch plus the startswith(LEGACY_AUTH_ERROR_MESSAGE_PREFIX) fallback, which is what keeps the pinned scratchpad pods' ConnectionError mapping to provider_auth;
• the capability gate at :1164 / :1208 — supports_api_key_provider from inspect.signature, with the warn-once degradation.

The only delta in turn_errors.py is 24 deletions of the WORKER_UNRESPONSIVE block. That is not a removal by this PR: grep on origin/main returns 0 hits and origin/staging returns 2, so it is ENG-2126 work that exists on staging and not on main. Correct for a main-based branch, and worth knowing that it means production keeps conflating "the worker never answered" with "anton raised something we don't recognise" until #427 reaches main by its own route.

The one blocking step

pyproject.toml:102 pins anton-agent to rev = "b4471472…" — anton#429's head, which is not an ancestor of anton main (verified). The comment says exactly this and instructs the re-pin. Restating because it is now a production build: merge anton#429 first, re-pin to the resulting anton main commit or branch = "main", regenerate the lockfile, then merge this. The image installs via uv sync --frozen, so a rev on an unmerged branch reaching a cowork-server main build is a broken image, not a stale one.

Two things that apply to the whole hotfix set

Six open PRs now exist for ENG-2116 — anton#421 / cowork-server#428 / cowork#778 against staging (all three approved), and anton#429 / cowork-server#444 / this set against main. Both tracks carry the same change. Whichever ships, the other wants closing or rebasing deliberately, and per the repo convention main → staging gets merged back by hand — nothing does it automatically, so staging will otherwise sit behind production carrying an approved duplicate of what already shipped.

main is production. These merge straight to it, so the usual staging soak does not happen. That raises the cost of the pin step below from "housekeeping" to "the thing that breaks the build".

---

Verdict: COMMENT. No new findings — the content is what I already approved. Not an approval; the pin step is a real precondition and it is Alejandro's call whether it is satisfied.

[alecantu7]

Approving

Head unchanged from the one I reviewed (202fd3b4), all checks complete with zero failures.

This carries the content I approved on cowork-server#428 across four rounds. Verified by diffing the ENG-2116 files rather than re-deriving: responses.py, providers.py, health.py and test_build_llm_client.py are byte-identical. Both findings I raised survive — the hosted-401 mapping at :950-954 (typed branch plus the startswith legacy fallback that keeps the pinned scratchpad pods mapping to provider_auth), and the capability gate at :1164/:1208 with its warn-once degradation.

The only turn_errors.py delta is 24 deletions of the WORKER_UNRESPONSIVE block, which is not a removal by this PR: grep returns 0 hits on origin/main and 2 on origin/staging. It is ENG-2126 work that has not reached main. Correct for a main-based branch — with the consequence that production keeps conflating "the worker never answered" with "anton raised something unrecognised" until #427 arrives by its own route.

The pin sequence is a precondition on merge, not on this approval. cowork-server#444 pins anton to a rev on an unmerged branch, and its own comment says so. Order: merge anton#429 → re-pin #444 to the resulting anton main commit (or branch = "main") → regenerate the lockfile → merge #444 → merge cowork#806. The image installs via uv sync --frozen, so a rev on an unmerged branch reaching a cowork-server main build is a broken image, not a stale one.\n\nBoth tracks are still open. anton#421 / cowork-server#428 / cowork#778 target staging and are also approved. Whichever ships, the other wants closing or rebasing deliberately — main → staging is a manual merge here.

Approved on Alejandro's call. Merging is his — and this one is the step the pin sequence is about.

[alecantu7]

Correction to my approval above. It says "head unchanged from the one I reviewed (202fd3b4)". That was true when I checked and not when I posted — the branch moved to 03c14b80 in between, so the approval is recorded against a head that claim does not describe. Flagging it rather than leaving a wrong statement in an approval.

I have now verified the two commits that landed, and both are fine — better than fine, in one case:

• 333eb2c3 adds docs plus a comment on the role == "router" branch explaining it is defensive and unreachable today, with the three swallowing call sites named (summarize(), gate() inside _gate_turn, _route_decision). That is exactly the note I suggested in my remainder review, so the next mutation sweep will not file it as a coverage gap. A test came with it.
• 03c14b80 bumps the anton pin to 303bf7c6 — anton#429's current head.

That second one resolves the stale-pin half of what I flagged: the pin now tracks its sibling instead of a superseded commit. The other half stands unchanged and is still the merge precondition — 303bf7c6 is not an ancestor of anton main, so it must be re-pinned to the resulting anton main commit (or branch = "main") with the lockfile regenerated, after anton#429 merges and before this one does.

The approval stands on 03c14b80.