Skip to content

chore: give ad-hoc Mintlify agent runs an instruction surface for our doc conventions #390

Description

@dangrondahl

When someone asks the Mintlify agent to write documentation — from Slack, or any trigger other than a .mintlify/workflows/ file — nothing in this repo tells it our conventions. The scheduled changelog workflow has a prompt; ad-hoc runs have none.

skill.md is not the mechanism for this

Worth stating up front, because it is the obvious first guess. Per Mintlify's skill.md docs, skill.md is an open standard for publishing your product's capabilities so that external agents can consume them, served from discovery endpoints (/.well-known/agent-skills/, /.well-known/skills/) and installable with npx skills add <url>. It documents Kosli to other people's agents.

It is not an instruction surface for the agent that edits this repo, and the docs say nothing about it applying to the doc-writing agent or the Slack integration. Auto-generation also takes up to 24 hours and only runs for public sites. A skill.md describing "how to write Kosli docs" would be published to the world as a Kosli product capability — wrong audience, wrong effect.

(Publishing a real skill.md for the Kosli CLI and API is a genuinely interesting idea, but it is a product-marketing decision, not this problem. Separate issue if we want it.)

What the actual surfaces appear to be

  1. .mintlify/workflows/*.md — per Mintlify's workflow docs, YAML frontmatter plus a Markdown prompt, one trigger per file. This is what update-changelog.md uses. Covers triggered runs only.
  2. CLAUDE.md — Mintlify's own guidance says a CLAUDE.md can "define house style, front matter rules, and Git practices so that agent-written updates stay consistent". If that applies to their agent on every run, it is the right home for standing instructions, and we already have a good one.

This needs verifying before any work is planned, and the docs do not state it: I could not confirm from Mintlify's documentation whether their agent reads CLAUDE.md, whether it reads .claude/skills/, or what it loads on a Slack-triggered run. The agent docs say only that it "reads your project content and any connected repositories".

Evidence that something is missing

#382 was a Slack-requested run (Requested by: … via Slack, Mintlify session: slack_…). It got the mechanical rules right — three pages created, all three added to config/navigation.json, root-relative links throughout, a conventional-commit title, mostly sentence-case headings. So the agent is not ignoring our conventions wholesale.

What it missed is the judgment layer that .claude/skills/doc-write/SKILL.md encodes:

  • Its PR description materially misdescribes its own diff. It says "Adds three placeholder pages for content to be written manually" and "Each page contains only its title". All three pages have substantial written content. A reviewer trusting the description approves it as empty scaffolding.
  • It did not check its pages against the pages they contradict. automated_user_provisioning.md states that a provisioned user's roles cannot be changed in Kosli, while roles_in_kosli.md — which it does edit — still asserts unconditionally that Admins can invite, remove, and change roles, in three places it left untouched. doc-write's final writing step is exactly this check: "does the change contradict a generated reference page, a schema, or a changelog entry?" Tracked in Document SCIM user provisioning under administration/managing_users #385.
  • No Diátaxis classification or placement rationale, which doc-write requires and which is where its "Decide where the page goes" table earns its keep.

Separately, on the workflow that does have a prompt: 7 of 10 changelog PRs used chore: where .mintlify/workflows/update-changelog.md explicitly says to use docs:. So even an explicit written instruction is not reliably followed — relevant to how much any of this can be leaned on. See #389.

Proposed work

  1. Establish what the agent actually reads. Ask Mintlify support, or test it: put a distinctive, harmless instruction in CLAUDE.md and make an ad-hoc request that would violate it. Everything below depends on the answer.
  2. If it reads CLAUDE.md — move the doc-writing rules that matter most into it, or have it point at .claude/skills/doc-write/SKILL.md explicitly. Keep one source of truth; do not fork the conventions into a third file that drifts.
  3. If it does not — add a .mintlify/workflows/ entry for on-demand doc writing whose prompt defers to CLAUDE.md and doc-write, so Slack requests have a path that carries the conventions. Worth confirming whether a workflow can be triggered on demand rather than only by cron.
  4. Either way, add the things docs: add blank authentication and user provisioning pages #382 actually missed, since they are cheap and specific:
    • The PR description must describe the diff as written. If pages were requested as placeholders and came out with content, say so.
    • Before opening the PR, grep for pages that state the opposite of what the new page states, and fix or flag them.
    • State the Diátaxis type and the chosen tab/group, with one line on why.
    • Conventional Commits for the PR title, per CLAUDE.md core rule 4.

Out of scope

Rewriting .claude/skills/doc-write/SKILL.md. It is in good shape and is the thing to point at, not to duplicate.

Open questions

  • Does the Mintlify agent read CLAUDE.md on every run, including Slack-triggered ones?
  • Does it read .claude/skills/, or is that Claude Code only?
  • Can a .mintlify/workflows/ file be triggered on demand, or is on: limited to cron and event triggers?
  • Is there a Mintlify-side setting for standing agent instructions that does not live in the repo?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automationWorkflow / CI automation tasksdiscoveryExploratory / requires Mintlify ProdocumentationImprovements or additions to documentationdxDeveloper experience, tooling, process

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions