(bug) Model picker shows a false "not available" badge, and Bedrock credential failures hang instead of failing fast - Fixed - #5522
Conversation
…redential failures hang instead of failing fast - fixed
dosubot
Bot
added
the
size:L
|
@mannietech15 is attempting to deploy a commit to the agenta projects Team on Vercel. A member of the Team first needs to authorize it. |
|
|
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe PR expands contribution guidance, adds a script for repairing local setup issues, and updates model-harness compatibility checks to account for custom-provider secrets and connection slugs, with regression tests. ChangesContribution guidance
Local setup repair
Custom-provider model reachability
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 628c99f8-fa2c-4ae2-a5ff-5401798d0666
📒 Files selected for processing (5)
CONTRIBUTING.mdfix-agenta-setup.shweb/packages/agenta-entity-ui/src/DrillInView/SchemaControls/agentTemplate/useModelHarness.tsxweb/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.tsweb/packages/agenta-entity-ui/tests/unit/connectionUtils.test.ts
| ## Quickstart & Local Development | ||
|
|
||
| 1. **Clone the Repository**: | ||
| ```bash | ||
| git clone https://github.com/agenta-ai/agenta.git | ||
| cd agenta | ||
| ``` | ||
|
|
||
| 2. **Install Dependencies**: | ||
| ```bash | ||
| pnpm install | ||
| ``` | ||
|
|
||
| 3. **Start Development Server**: | ||
| ```bash | ||
| pnpm dev | ||
| ``` | ||
|
|
||
| 4. **Lint & Test**: | ||
| ```bash | ||
| pnpm lint | ||
| pnpm test | ||
| ``` | ||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
python - <<'PY'
import json
from pathlib import Path
for path in [Path("package.json"), Path("web/package.json")]:
if path.exists():
data = json.loads(path.read_text())
print(path, sorted(data.get("scripts", {})))
PYRepository: Agenta-AI/agenta
Length of output: 517
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "tracked manifests/workspaces:"
fd -a 'package\.json$' . | sed 's#^\./##' | sort
echo
echo "root package.json:"
if [ -f package.json ]; then cat -n package.json | sed -n '1,160p'; fi
echo
echo "related docs and workflow references:"
for f in CONTRIBUTING.md README.md; do
if [ -f "$f" ]; then
echo "--- $f lines 1-90 ---"
cat -n "$f" | sed -n '1,90p'
fi
done
echo
echo "search pnpm package scripts references:"
rg -n "pnpm (install|dev|lint|test|install:(workspace-)?force)|test:apps|dev-oss|dev-ee" -S .github CONTRIBUTING.md README.md package.json web/package.json 2>/dev/null || trueRepository: Agenta-AI/agenta
Length of output: 11742
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "root pnpm files:"
fd -a '^(pnpm-workspace\.yaml|pnpm-lock\.yaml)$' . | sed 's#^\./##'
echo
echo "root tracked files around top-level manifests/workspaces:"
git ls-files | awk '/(^|\/)(package\.json|pnpm-workspace\.yaml|pnpm-lock\.yaml|package-lock\.json|yarn\.lock|bun\.lockb|bun\.lock)$/{print}' | head -100
echo
echo "web/package.json relevant scripts:"
python3 - <<'PY'
import json
from pathlib import Path
p = Path("web/package.json")
data = json.loads(p.read_text())
for k in ["scripts", "private", "repository", "packageManager"]:
print(k, data.get(k))
PY
echo
echo "test workflow command references:"
while IFS=: read -r file line rest; do
[ -z "$file" ] && continue
echo "--- ${file}:${line} ---"
sed -n "$((line-6)),$((line+10))p" "$file"
done < <(rg -n "cd web|pnpm (install|lint|test|.*-oss|.*-ee)|dev-oss|dev-ee|test:apps|turbo run|pnpm test" .github/workflows -S)Repository: Agenta-AI/agenta
Length of output: 4006
Use web workspace commands in the quickstart and PR checklist.
There is no root package.json or root pnpm install, while the web workspace defines dev-oss/dev-ee, lint, and test:datalayer under web/. Update lines 19-42 and 51-60 to document the correct cd web && pnpm … commands, such as pnpm dev-oss and pnpm test:datalayer.
| sudo tee /etc/docker/daemon.json > /dev/null <<'EOF' | ||
| { | ||
| "ipv6": false, | ||
| "dns": ["8.8.8.8", "8.8.4.4"] | ||
| } | ||
| EOF | ||
| echo "Restarting Docker daemon..." | ||
| sudo systemctl restart docker | ||
| sleep 3 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Do not overwrite the entire Docker daemon configuration.
This heredoc deletes any existing Docker settings, such as registry mirrors, storage configuration, logging, proxies, or runtime options. Merge only the intended DNS/IPv6 changes, preserve a backup, validate the resulting JSON, and restart only after validation succeeds.
| echo "=== Fix 3: Verify env files ===" | ||
| [[ -f "hosting/docker-compose/oss/.env.oss.dev" ]] && echo "✓ .env.oss.dev exists" || echo "✗ .env.oss.dev MISSING" | ||
| [[ -f "hosting/docker-compose/oss/.env.oss.gh" ]] && echo "✓ .env.oss.gh exists" || echo "✗ .env.oss.gh MISSING" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Fail when no usable environment file exists.
These checks only print warnings, so the script continues to advertise commands that will fail later. Also, .env.oss.dev and .env.oss.gh are alternatives for different compose modes, not universally required files. Validate the selected mode, or exit nonzero when neither file exists.
Suggested validation
+dev_env="hosting/docker-compose/oss/.env.oss.dev"
+gh_env="hosting/docker-compose/oss/.env.oss.gh"
+
+if [[ ! -f "$dev_env" && ! -f "$gh_env" ]]; then
+ echo "✗ No usable environment file found"
+ exit 1
+fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| echo "=== Fix 3: Verify env files ===" | |
| [[ -f "hosting/docker-compose/oss/.env.oss.dev" ]] && echo "✓ .env.oss.dev exists" || echo "✗ .env.oss.dev MISSING" | |
| [[ -f "hosting/docker-compose/oss/.env.oss.gh" ]] && echo "✓ .env.oss.gh exists" || echo "✗ .env.oss.gh MISSING" | |
| dev_env="hosting/docker-compose/oss/.env.oss.dev" | |
| gh_env="hosting/docker-compose/oss/.env.oss.gh" | |
| if [[ ! -f "$dev_env" && ! -f "$gh_env" ]]; then | |
| echo "✗ No usable environment file found" | |
| exit 1 | |
| fi | |
| echo "=== Fix 3: Verify env files ===" | |
| [[ -f "hosting/docker-compose/oss/.env.oss.dev" ]] && echo "✓ .env.oss.dev exists" || echo "✗ .env.oss.dev MISSING" | |
| [[ -f "hosting/docker-compose/oss/.env.oss.gh" ]] && echo "✓ .env.oss.gh exists" || echo "✗ .env.oss.gh MISSING" |
| if (hasCatalog && catalog!.some((e) => e.id === modelId)) return true | ||
| if ( | ||
| hasModels && | ||
| Object.values(models!).some((ids) => Array.isArray(ids) && ids.includes(modelId)) | ||
| ) | ||
| return true | ||
|
|
||
| if (customSecrets?.length) { | ||
| for (const secret of customSecrets) { | ||
| const secretSlug = secret.name?.trim() | ||
| const kind = secret.provider?.toLowerCase() || null | ||
| const secretModels = (secret.models ?? []).filter(Boolean) | ||
| if (!secretModels.includes(modelId)) continue | ||
| if (slug && secretSlug !== slug) continue | ||
| if (!kind || harnessReachesCustomProviderKind(capabilities, harness, kind)) return true |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Prioritize the selected custom connection over generic model/provider matches.
With a connection slug, Lines 369-374 can accept a colliding catalog model before validating that connection, and Line 529 can accept an unreachable deployment through its provider family. For example, a bedrock connection using an Anthropic-shaped model ID appears supported on pi_core because it supports anthropic, despite not consuming Bedrock.
web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.ts#L369-L383: whenslugis present, resolve the exact vault secret and return its deployment reachability before checking generic catalog/models.web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/agentTemplate/useModelHarness.tsx#L526-L529: only apply the provider-family fallback whenconnection.slugis absent.web/packages/agenta-entity-ui/tests/unit/connectionUtils.test.ts#L260-L329: add a colliding-ID test and a selected-status assertion for a slug-bound Bedrock connection on a harness that supports the model family but not Bedrock.
Proposed direction
+ if (slug) {
+ const secret = customSecrets?.find(
+ (candidate) =>
+ candidate.name?.trim() === slug &&
+ (candidate.models ?? []).includes(modelId),
+ )
+ if (!secret) return false
+ const kind = secret.provider?.toLowerCase() || null
+ return !kind || harnessReachesCustomProviderKind(capabilities, harness, kind)
+ }
+
if (hasCatalog && catalog!.some((e) => e.id === modelId)) return true- (!!connection.provider && selectedProviders.includes(connection.provider))
+ (!connection.slug &&
+ !!connection.provider &&
+ selectedProviders.includes(connection.provider))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if (hasCatalog && catalog!.some((e) => e.id === modelId)) return true | |
| if ( | |
| hasModels && | |
| Object.values(models!).some((ids) => Array.isArray(ids) && ids.includes(modelId)) | |
| ) | |
| return true | |
| if (customSecrets?.length) { | |
| for (const secret of customSecrets) { | |
| const secretSlug = secret.name?.trim() | |
| const kind = secret.provider?.toLowerCase() || null | |
| const secretModels = (secret.models ?? []).filter(Boolean) | |
| if (!secretModels.includes(modelId)) continue | |
| if (slug && secretSlug !== slug) continue | |
| if (!kind || harnessReachesCustomProviderKind(capabilities, harness, kind)) return true | |
| if (slug) { | |
| const secret = customSecrets?.find( | |
| (candidate) => | |
| candidate.name?.trim() === slug && | |
| (candidate.models ?? []).includes(modelId), | |
| ) | |
| if (!secret) return false | |
| const kind = secret.provider?.toLowerCase() || null | |
| return !kind || harnessReachesCustomProviderKind(capabilities, harness, kind) | |
| } | |
| if (hasCatalog && catalog!.some((e) => e.id === modelId)) return true | |
| if ( | |
| hasModels && | |
| Object.values(models!).some((ids) => Array.isArray(ids) && ids.includes(modelId)) | |
| ) | |
| return true | |
| if (customSecrets?.length) { | |
| for (const secret of customSecrets) { | |
| const secretSlug = secret.name?.trim() | |
| const kind = secret.provider?.toLowerCase() || null | |
| const secretModels = (secret.models ?? []).filter(Boolean) | |
| if (!secretModels.includes(modelId)) continue | |
| if (slug && secretSlug !== slug) continue | |
| if (!kind || harnessReachesCustomProviderKind(capabilities, harness, kind)) return true |
📍 Affects 3 files
web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/connectionUtils.ts#L369-L383(this comment)web/packages/agenta-entity-ui/src/DrillInView/SchemaControls/agentTemplate/useModelHarness.tsx#L526-L529web/packages/agenta-entity-ui/tests/unit/connectionUtils.test.ts#L260-L329
2 participants
Summary
The "model not available" compatibility badge in the harness picker incorrectly flagged
vault-managed custom-provider models as unavailable, even when the backend accepted and ran them
successfully.
harnessAllowsModelinconnectionUtils.tsalready acceptscustomSecretsandslugparametersto check vault connections before concluding a model is unreachable. The function itself was correct
and unit-tested. The bug was at the call site in
useModelHarness.tsx:selectedKeepsModelcalledharnessAllowsModelwithout those arguments, so the vault path was never consulted.Before:
After:
Both
customSecrets(fromcustomSecretsAtom) andconnection.slug(decoded fromconfig.llm)were already in scope at the call site; they just weren't being passed through.
Testing
Verified locally
pnpm testinweb/packages/agenta-entity-ui— 12 test files, 210 tests, all passed.Added or updated tests
Added
"selectedKeepsModel regression: vault model flagged unavailable without secrets, available with them"toconnectionUtils.test.ts. The test explicitly asserts the broken pre-fix behaviour(no secrets passed →
false) against the fixed behaviour (secrets + slug passed →true) andincludes a slug-mismatch case (a credential for a different connection must not grant availability).
QA follow-up
encode a recognizable provider family (for example a bare slug like
custom-bedrock-model-id-123).it showed "model not available" in amber.
under their respective harnesses.
Demo
N/A — badge label and colour change only, no structural UI change. The QA steps above describe
where to look.
Checklist
Contributor Resources
agenta-2.mp4