feat(docs): persist the docs LLM agent choice (config + onboarding + CLI)#312
Merged
Conversation
Persist which host CLI authors the docs in ~/.deeplake/config.json (docs.llmAgent). Resolution precedence in resolveDocLlmSpec becomes: HIVEMIND_DOCS_LLM_BIN > HIVEMIND_DOCS_LLM_AGENT (env) > config > auto-detect so the env override stays a one-off knob while the persisted choice survives restarts. Adds detectAvailableAgents()/knownDocsAgents() helpers and a `hivemind docs agent [name]` command (show current + installed, or set with validation; no org creds needed). Sonnet remains the slow-tier default.
runDocsOnboarding now asks "Which agent should write the docs?" — but only when >1 host CLI is installed AND none is pinned yet; one agent (or an existing pin) stays silent. The choice persists globally via config.json. detect/get/set are injectable for tests. Updates the existing onboarding tests to opt out of the new gate (detectAgents: () => []) so they stay deterministic regardless of what's installed on the test host.
📝 WalkthroughWalkthroughAdds persistent docs LLM agent configuration, automatic agent discovery, onboarding selection when multiple agents are installed, and the ChangesDocs LLM agent selection
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant runDocsOnboarding
participant detectAvailableAgents
participant OnboardingIo
participant setDocsLlmAgent
User->>runDocsOnboarding: accept docs generation
runDocsOnboarding->>detectAvailableAgents: detect installed agents
detectAvailableAgents-->>runDocsOnboarding: available agents
runDocsOnboarding->>OnboardingIo: ask for agent choice
OnboardingIo-->>runDocsOnboarding: selected or blank answer
runDocsOnboarding->>setDocsLlmAgent: persist selected agent
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Coverage ReportScope: files changed in this PR. Enforced threshold: 90% per metric (per file via
File Coverage — 4 files changed
Generated for commit 47a0854. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/shared/docs-llm-agent.test.ts (1)
144-145: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAssert on exact messages rather than using generic regex substrings.
As per path instructions for
tests/**: "Prefer asserting on specific values (paths, messages) over generic substrings." The assertions against theaskedarray currently use regex partial matches. These should be refactored to verify the exact prompt strings or the entireaskedarray.♻️ Proposed refactor
- expect(res.generate).toBe(true); - expect(asked.some((q) => /Which agent should write the docs/.test(q))).toBe(true); + expect(res.generate).toBe(true); + expect(asked).toContain("Which agent should write the docs? [claude/codex] (default: claude) ");And for the negative assertions, you can verify the exact array contents for better strictness:
Line 172:
- expect(asked.some((q) => /Which agent/.test(q))).toBe(false); + expect(asked).toEqual([ + "Generate wiki docs for this repo now? (one-time LLM cost) [y/N] ", + "Keep them automatically in sync on every commit? Docs stay fresh but this consumes more LLM tokens over time. [y/N] " + ]);Line 186:
- expect(asked.some((q) => /Which agent/.test(q))).toBe(false); + expect(asked).toEqual([ + "Generate wiki docs for this repo now? (one-time LLM cost) [y/N] ", + "Keep them automatically in sync on every commit? Docs stay fresh but this consumes more LLM tokens over time. [y/N] " + ]);Line 200:
- expect(asked.some((q) => /Which agent/.test(q))).toBe(false); + expect(asked).toEqual([ + "Generate wiki docs for this repo now? (one-time LLM cost) [y/N] " + ]);Also applies to: 172-172, 186-186, 200-200
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/shared/docs-llm-agent.test.ts` around lines 144 - 145, Replace the generic regex-based assertions on the asked array in this test with exact prompt-string or full-array equality checks. Update the related assertions at the other referenced cases, including the negative cases, to verify the complete expected asked contents rather than partial substrings while preserving the existing generate-result checks.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/shared/docs-llm-agent.test.ts`:
- Around line 144-145: Replace the generic regex-based assertions on the asked
array in this test with exact prompt-string or full-array equality checks.
Update the related assertions at the other referenced cases, including the
negative cases, to verify the complete expected asked contents rather than
partial substrings while preserving the existing generate-result checks.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 5d2b097f-7039-4452-86fa-7ea731bb4933
📒 Files selected for processing (6)
src/commands/docs.tssrc/docs/onboarding.tssrc/docs/refresh-llm.tssrc/user-config.tstests/shared/docs-llm-agent.test.tstests/shared/docs-onboarding.test.ts
The new `hivemind docs agent` handler added uncovered branches to commands/docs.ts, dropping its branch coverage below the 85% gate. Exercise show (unset + pinned), set (case-insensitive persist), and the unknown-agent rejection (process.exit) via runDocsCommand, restoring the threshold.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/shared/docs-llm-agent.test.ts`:
- Around line 233-255: Update the tests around runDocsCommand to assert exact
expected log or error messages and specific log array elements, replacing
generic regex/substring checks on logs.join("\n"). Preserve the existing
behavioral coverage for default, pinned, case-insensitive set, and unknown-agent
flows, including the persistence and process-exit assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 326b66f5-faee-4c65-a66e-ab3aab8b47b4
📒 Files selected for processing (1)
tests/shared/docs-llm-agent.test.ts
Comment on lines
+233
to
+255
| it("no arg → shows current (auto when unset) + installed + how to set", async () => { | ||
| await runDocsCommand(["agent"]); | ||
| expect(logs.join("\n")).toMatch(/Docs LLM agent:/); | ||
| expect(logs.join("\n")).toMatch(/Set with: hivemind docs agent/); | ||
| }); | ||
|
|
||
| it("no arg after a pin → shows the pinned agent", async () => { | ||
| setDocsLlmAgent("codex"); | ||
| await runDocsCommand(["agent"]); | ||
| expect(logs.join("\n")).toContain("Docs LLM agent: codex"); | ||
| }); | ||
|
|
||
| it("set a known agent → persists it", async () => { | ||
| await runDocsCommand(["agent", "Codex"]); // case-insensitive | ||
| expect(getDocsLlmAgent()).toBe("codex"); | ||
| expect(logs.join("\n")).toContain("Docs LLM agent set to: codex"); | ||
| }); | ||
|
|
||
| it("unknown agent → error + non-zero exit, nothing persisted", async () => { | ||
| await expect(runDocsCommand(["agent", "bogus"])).rejects.toThrow("process.exit"); | ||
| expect(errs.join("\n")).toMatch(/Unknown agent/); | ||
| expect(getDocsLlmAgent()).toBeUndefined(); | ||
| }); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Assert on specific messages rather than generic substrings.
As per path instructions, prefer asserting on exact messages and array elements instead of using .join("\n") with generic substring checks.
♻️ Proposed fixes
it("no arg → shows current (auto when unset) + installed + how to set", async () => {
await runDocsCommand(["agent"]);
- expect(logs.join("\n")).toMatch(/Docs LLM agent:/);
- expect(logs.join("\n")).toMatch(/Set with: hivemind docs agent/);
+ expect(logs[0]).toMatch(/^Docs LLM agent: auto \(/);
+ expect(logs[2]).toBe("Set with: hivemind docs agent <claude|codex|pi|cursor> (or HIVEMIND_DOCS_LLM_BIN for any other CLI)");
});
it("no arg after a pin → shows the pinned agent", async () => {
setDocsLlmAgent("codex");
await runDocsCommand(["agent"]);
- expect(logs.join("\n")).toContain("Docs LLM agent: codex");
+ expect(logs[0]).toBe("Docs LLM agent: codex");
});
it("set a known agent → persists it", async () => {
await runDocsCommand(["agent", "Codex"]); // case-insensitive
expect(getDocsLlmAgent()).toBe("codex");
- expect(logs.join("\n")).toContain("Docs LLM agent set to: codex");
+ expect(logs[0]).toBe("Docs LLM agent set to: codex. Change with: hivemind docs agent <name>.");
});
it("unknown agent → error + non-zero exit, nothing persisted", async () => {
await expect(runDocsCommand(["agent", "bogus"])).rejects.toThrow("process.exit");
- expect(errs.join("\n")).toMatch(/Unknown agent/);
+ expect(errs[0]).toBe('Unknown agent "bogus". Known: claude, codex, pi, cursor. For any other CLI use HIVEMIND_DOCS_LLM_BIN.');
expect(getDocsLlmAgent()).toBeUndefined();
});📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| it("no arg → shows current (auto when unset) + installed + how to set", async () => { | |
| await runDocsCommand(["agent"]); | |
| expect(logs.join("\n")).toMatch(/Docs LLM agent:/); | |
| expect(logs.join("\n")).toMatch(/Set with: hivemind docs agent/); | |
| }); | |
| it("no arg after a pin → shows the pinned agent", async () => { | |
| setDocsLlmAgent("codex"); | |
| await runDocsCommand(["agent"]); | |
| expect(logs.join("\n")).toContain("Docs LLM agent: codex"); | |
| }); | |
| it("set a known agent → persists it", async () => { | |
| await runDocsCommand(["agent", "Codex"]); // case-insensitive | |
| expect(getDocsLlmAgent()).toBe("codex"); | |
| expect(logs.join("\n")).toContain("Docs LLM agent set to: codex"); | |
| }); | |
| it("unknown agent → error + non-zero exit, nothing persisted", async () => { | |
| await expect(runDocsCommand(["agent", "bogus"])).rejects.toThrow("process.exit"); | |
| expect(errs.join("\n")).toMatch(/Unknown agent/); | |
| expect(getDocsLlmAgent()).toBeUndefined(); | |
| }); | |
| it("no arg → shows current (auto when unset) + installed + how to set", async () => { | |
| await runDocsCommand(["agent"]); | |
| expect(logs[0]).toMatch(/^Docs LLM agent: auto \(/); | |
| expect(logs[2]).toBe("Set with: hivemind docs agent <claude|codex|pi|cursor> (or HIVEMIND_DOCS_LLM_BIN for any other CLI)"); | |
| }); | |
| it("no arg after a pin → shows the pinned agent", async () => { | |
| setDocsLlmAgent("codex"); | |
| await runDocsCommand(["agent"]); | |
| expect(logs[0]).toBe("Docs LLM agent: codex"); | |
| }); | |
| it("set a known agent → persists it", async () => { | |
| await runDocsCommand(["agent", "Codex"]); // case-insensitive | |
| expect(getDocsLlmAgent()).toBe("codex"); | |
| expect(logs[0]).toBe("Docs LLM agent set to: codex. Change with: hivemind docs agent <name>."); | |
| }); | |
| it("unknown agent → error + non-zero exit, nothing persisted", async () => { | |
| await expect(runDocsCommand(["agent", "bogus"])).rejects.toThrow("process.exit"); | |
| expect(errs[0]).toBe('Unknown agent "bogus". Known: claude, codex, pi, cursor. For any other CLI use HIVEMIND_DOCS_LLM_BIN.'); | |
| expect(getDocsLlmAgent()).toBeUndefined(); | |
| }); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/shared/docs-llm-agent.test.ts` around lines 233 - 255, Update the tests
around runDocsCommand to assert exact expected log or error messages and
specific log array elements, replacing generic regex/substring checks on
logs.join("\n"). Preserve the existing behavioral coverage for default, pinned,
case-insensitive set, and unknown-agent flows, including the persistence and
process-exit assertions.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Make the docs-generation host agent a persistent choice instead of an env var that vanishes when you close the shell. You asked: "why does it default to claude, and where do I change it so it survives a restart?"
Behaviour
docs.llmAgentin~/.deeplake/config.json(survives restarts, machines, sessions).resolveDocLlmSpec:HIVEMIND_DOCS_LLM_BIN>HIVEMIND_DOCS_LLM_AGENT(env, one-off) > configdocs.llmAgent> auto-detect (claude → codex → pi → cursor).runDocsOnboarding) — but only when >1 host CLI is installed AND none is pinned yet. One agent, or an existing pin → no question (zero nag). The choice persists.hivemind docs agent [name]— show current + installed, or set (validated; warns if not on PATH; no org creds needed).Out of scope (explicitly deferred): a persistent per-tier model override, and giving
cursor/pithe fast/slow split. Sonnet stays the slow-tier default.Verification
tscclean; 20 unit tests (precedence env>config>auto, config persistence + no-clobber ofembeddings, install detection, onboarding gate across all cases: >1/1/already-pinned/declined).docs agent cursor, process B (fresh)docs agentreadscursorback from disk → survives restart. Unknown agent → exit 1.tree-sitter absentbundle test, an env artifact on Node 24 that passes in CI).Files
src/user-config.ts—docs.llmAgentfield +get/setDocsLlmAgentsrc/docs/refresh-llm.ts— precedence +detectAvailableAgents/knownDocsAgentssrc/commands/docs.ts—docs agentcommand + USAGEsrc/docs/onboarding.ts— the gated agent question (injectable deps)tests/shared/docs-llm-agent.test.ts(new),tests/shared/docs-onboarding.test.ts(opt out of the new gate)Session Context
Session transcript
Summary by CodeRabbit
hivemind docs agentto view the currently configured agent (or available options) and to set it for future runs.