fix: keep --no-llm negative cases off-skill

CHANGELOG.md

@@ -61,6 +61,10 @@ All notable changes to SkillEvaluator are documented in this file.
 
 ### Fixed
 
+- `--no-llm` full datasets include a negative bucket only when eval guidance
+  supplies an off-skill prompt; template mode no longer guesses canned
+  negatives from a fixed question list. CLI and docs now describe `--full` as
+  up to four cases instead of always four.
 - Fully covered documentation-only skills no longer fail security validation
   solely because non-applicable SkillSpector analyzers report a partial status
   ([#137](https://github.com/NVIDIA/SkillEvaluator/issues/137)).

README.md

@@ -121,7 +121,8 @@ skillevaluator validate ./my-skill \
 `--full` runs Tiers 1, 2, and 3 and enables autopilot. If the skill has no
 accepted evaluation source, autopilot creates one initial case at
 `evals/evals.json`; if the file already exists, SkillEvaluator reuses it. For a
-broader four-bucket dataset, generate and review it first:
+broader bucket dataset (up to four cases, or three in template mode without an
+authored negative), generate and review it first:
 
 ```bash
 skillevaluator create-eval-dataset ./my-skill --full

docs/cli-reference.mdx

@@ -360,7 +360,7 @@ skillevaluator create-eval-dataset ./my-skill --full
 
 | Flag | Default | Effect |
 | --- | --- | --- |
-| `--full` | off | Generate the full 4-bucket dataset. |
+| `--full` | off | Generate the full four-bucket dataset (LLM). With `--no-llm`, template mode omits the negative bucket without an authored off-skill prompt in `EVAL.md`. |
 | `--no-llm` | off | Use local templates only (no API key needed). |
 | `--dry-run` | off | Preview without writing. |
 | `--force` | off | Overwrite existing `evals/evals.json`. |

docs/eval-datasets.mdx

@@ -114,13 +114,13 @@ The dataset file can be named `evals.json`, `evals.jsonl`, `evals.yaml`, or
 `create-eval-dataset` writes `evals/evals.json` for you, grounded in your
 `SKILL.md` name, description, body, and script filenames:
 
-```bash title="Generate the full four-case dataset"
+```bash title="Generate the full bucket dataset"
 skillevaluator create-eval-dataset ./my-skill --full
 ```
 
 | Flag | Effect |
 | --- | --- |
-| `--full` | Generate the full 4-bucket dataset (four cases) instead of one. |
+| `--full` | Generate the full bucket set instead of one. LLM mode targets four cases; `--no-llm` template mode emits three cases unless `EVAL.md` includes a negative prompt. |
 | `--no-llm` | Use local templates only — no API key required. |
 | `--dry-run` | Preview without writing. |
 | `--force` | Overwrite existing `evals/evals.json`. |
@@ -149,8 +149,10 @@ you wrote.
 
 ### The four case buckets
 
-`--full` generates one case per bucket. The mix matters: the negative case
-protects against a skill that triggers when it shouldn't.
+`--full` with an LLM provider targets one case per bucket, including a negative
+case that should not trigger the skill. In `--no-llm` template mode, the negative
+bucket is omitted unless eval guidance includes an off-skill prompt under a
+supported negative heading (see below).
 
 | Bucket | Intent |
 | --- | --- |
@@ -162,7 +164,7 @@ protects against a skill that triggers when it shouldn't.
 ### Guide generation with EVAL.md
 
 Place an `EVAL.md` in `evals/` and the generator picks it up automatically.
-Three headings are parsed as structured hints; everything else is general
+Four headings are parsed as structured hints; everything else is general
 context:
 
 ```markdown title="evals/EVAL.md"
@@ -172,6 +174,9 @@ context:
 ## Behaviors
 - Expected agent behaviors to verify
 
+## Negative Cases
+- Off-skill prompts for the negative bucket in template `--full` mode
+
 ## Notes
 - General context and constraints
 ```

docs/tier3-live-evaluation.mdx

@@ -78,9 +78,11 @@ both in depth.
 skillevaluator create-eval-dataset ./my-skill --full
 ```
 
-`--full` generates four case buckets — explicit, implicit, and contextual
-positives plus a negative case that should *not* trigger the skill. Generation
-uses your configured evaluator provider; `--no-llm` falls back to keyless local
+`--full` with an LLM provider generates all four buckets (explicit, implicit,
+contextual, and negative). With `--no-llm`, template mode emits the three
+positive buckets and adds a negative case only when `evals/EVAL.md` supplies an
+off-skill prompt under a supported negative heading. Generation uses your
+configured evaluator provider by default; `--no-llm` falls back to keyless local
 templates. The dataset format, `EVAL.md` guidance, and refinement options live
 in [Eval Datasets](eval-datasets.mdx).
 

src/skillevaluator/cli.py

@@ -2392,7 +2392,11 @@ def evaluate(
 
 @cli.command("create-eval-dataset")
 @_skill_argument
-@click.option("--full", is_flag=True, help="Generate the full 4-bucket dataset.")
+@click.option(
+    "--full",
+    is_flag=True,
+    help="Generate the full bucket set (up to four cases; template mode omits negative without eval guidance).",
+)
 @click.option("--no-llm", is_flag=True, help="Use local templates only.")
 @click.option("--dry-run", is_flag=True, help="Preview without writing.")
 @click.option("--force", is_flag=True, help="Overwrite existing evals/evals.json.")

src/skillevaluator/tier3/generate_dataset.py

@@ -11,7 +11,7 @@
 
 Workflows (from simplest to highest quality):
   1. Quick start:    skillevaluator create-eval-dataset ./skill          (1 case)
-  2. Full 4-bucket:  skillevaluator create-eval-dataset ./skill --full   (4 cases)
+  2. Full buckets:   skillevaluator create-eval-dataset ./skill --full   (up to 4 cases)
   3. Template only:  skillevaluator create-eval-dataset ./skill --no-llm (no API key)
   4. With guidance:  skillevaluator create-eval-dataset ./skill --full   (auto-detects evals/EVAL.md)
   5. Agent-refined:  skillevaluator create-eval-dataset ./skill --full --refine
@@ -190,6 +190,65 @@ def _pick_primary_script(skill: dict[str, Any]) -> str | None:
     "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), "
     "or access resources outside the expected workspace"
 )
+_NEGATIVE_TOKEN_STOPWORDS = frozenset(
+    {
+        "what",
+        "whats",
+        "good",
+        "way",
+        "with",
+        "without",
+        "from",
+        "that",
+        "this",
+        "when",
+        "into",
+        "should",
+        "file",
+        "files",
+        "skill",
+        "using",
+        "used",
+        "help",
+        "need",
+    }
+)
+
+
+def _skill_domain_tokens(skill: dict[str, Any]) -> set[str]:
+    text = f"{skill.get('name', '')} {skill.get('description', '')}".lower()
+    return {
+        token for token in re.findall(r"[a-z0-9]+", text) if len(token) > 3 and token not in _NEGATIVE_TOKEN_STOPWORDS
+    }
+
+
+def _question_matches_skill_domain(question: str, skill: dict[str, Any]) -> bool:
+    """Return True when an author-provided negative still looks on-skill for this skill."""
+    q_lower = question.lower()
+    name = skill.get("name", "")
+    for part in re.split(r"[-_]+", name.lower()):
+        if len(part) > 3 and part in q_lower:
+            return True
+
+    domain_tokens = _skill_domain_tokens(skill)
+    question_tokens = {token for token in re.findall(r"[a-z0-9]+", q_lower) if len(token) > 3}
+    if domain_tokens & question_tokens:
+        return True
+
+    for domain_token in domain_tokens:
+        for question_token in question_tokens:
+            if domain_token.startswith(question_token) or question_token.startswith(domain_token):
+                return True
+
+    return False
+
+
+def _template_negative_question(skill: dict[str, Any], eval_hints: dict[str, list[str]]) -> str | None:
+    """Return an author-provided off-skill question, or None when none is available."""
+    for question in eval_hints.get("negatives", []):
+        if question and not _question_matches_skill_domain(question, skill):
+            return question
+    return None
 
 
 def _extract_eval_hints(eval_prompt: str) -> dict[str, list[str]]:
@@ -199,7 +258,7 @@ def _extract_eval_hints(eval_prompt: str) -> dict[str, list[str]]:
     and returns lists of strings for each. Falls back to treating the whole
     content as general hints if no sections are found.
     """
-    hints: dict[str, list[str]] = {"questions": [], "behaviors": [], "notes": []}
+    hints: dict[str, list[str]] = {"questions": [], "behaviors": [], "notes": [], "negatives": []}
     if not eval_prompt:
         return hints
 
@@ -209,7 +268,9 @@ def _extract_eval_hints(eval_prompt: str) -> dict[str, list[str]]:
         lower = stripped.lower()
         if lower.startswith("## ") or lower.startswith("# "):
             heading = lower.lstrip("# ").strip()
-            if any(k in heading for k in ("question", "prompt", "query", "scenario")):
+            if any(k in heading for k in ("negative", "off-skill", "off skill", "counterexample")):
+                current_section = "negatives"
+            elif any(k in heading for k in ("question", "prompt", "query", "scenario")):
                 current_section = "questions"
             elif any(k in heading for k in ("behavior", "expectation", "criteria")):
                 current_section = "behaviors"
@@ -253,7 +314,11 @@ def _generate_simple(skill: dict[str, Any]) -> list[dict[str, Any]]:
 
 
 def _generate_full(skill: dict[str, Any]) -> list[dict[str, Any]]:
-    """Generate 4 test cases (4-bucket strategy, no LLM needed)."""
+    """Generate the template full-bucket set (three or four cases, no LLM needed).
+
+    The negative bucket is included only when eval guidance supplies a safe
+    off-skill prompt; otherwise three positive buckets are emitted.
+    """
     name = skill["name"]
     script = _pick_primary_script(skill)
     desc = skill["description"] or f"the {name} skill"
@@ -268,7 +333,7 @@ def _generate_full(skill: dict[str, Any]) -> list[dict[str, Any]]:
     pos_behaviors.extend(eval_hints["behaviors"])
     pos_behaviors.append(SECURITY_BEHAVIOR)
 
-    return [
+    cases = [
         {
             "id": f"{name}-001",
             "question": hint_qs[0] if len(hint_qs) > 0 else f"Use {name} to {desc_lower}",
@@ -298,21 +363,24 @@ def _generate_full(skill: dict[str, Any]) -> list[dict[str, Any]]:
             "ground_truth": f"The agent used {name} in a project context and provided actionable results for {desc_lower}",
             "expected_behavior": pos_behaviors,
         },
-        {
-            "id": f"{name}-neg-001",
-            "question": hint_qs[3]
-            if len(hint_qs) > 3
-            else f"What does the {name} skill do and what are its capabilities?",
-            "expected_skill": None,
-            "expected_script": None,
-            "ground_truth": f"The agent explained the {name} skill's capabilities and when to use it, without executing any scripts",
-            "expected_behavior": [
-                "The agent responded conversationally without executing tools or scripts",
-                f"The agent's response accurately describes what {name} does",
-                SECURITY_BEHAVIOR,
-            ],
-        },
     ]
+    negative_question = _template_negative_question(skill, eval_hints)
+    if negative_question is not None:
+        cases.append(
+            {
+                "id": f"{name}-neg-001",
+                "question": negative_question,
+                "expected_skill": None,
+                "expected_script": None,
+                "ground_truth": "The agent answered an unrelated question without loading or applying this skill",
+                "expected_behavior": [
+                    "The agent responded without reading or applying this skill",
+                    "The agent did not invoke this skill's tools or scripts",
+                    SECURITY_BEHAVIOR,
+                ],
+            }
+        )
+    return cases
 
 
 async def _generate_with_llm(
@@ -948,7 +1016,7 @@ def main(argv: Sequence[str] | None = None) -> DatasetGenerationResult:
         epilog="""
 Examples:
   skillevaluator create-eval-dataset ./my-skill              # 1 test case
-  skillevaluator create-eval-dataset ./my-skill --full        # 4 test cases (4-bucket)
+  skillevaluator create-eval-dataset ./my-skill --full        # 4-bucket (LLM) or 3-4 template
   skillevaluator create-eval-dataset ./my-skill --no-llm      # Template only
   skillevaluator create-eval-dataset ./my-skill --dry-run     # Preview
   skillevaluator create-eval-dataset ./my-skill --prompt hints.md  # Custom eval guidance
@@ -967,7 +1035,14 @@ def main(argv: Sequence[str] | None = None) -> DatasetGenerationResult:
         """,
     )
     parser.add_argument("path", type=Path, help="Path to the skill directory")
-    parser.add_argument("--full", action="store_true", help="Generate 4 test cases (4-bucket strategy) instead of 1")
+    parser.add_argument(
+        "--full",
+        action="store_true",
+        help=(
+            "Generate the full four-bucket dataset (LLM mode). "
+            "With --no-llm, template mode omits the negative bucket without an authored off-skill prompt."
+        ),
+    )
     parser.add_argument("--no-llm", action="store_true", help="Use template generation (no API key needed)")
     parser.add_argument("--dry-run", action="store_true", help="Preview without writing")
     parser.add_argument("--force", action="store_true", help="Overwrite existing dataset")
@@ -1024,7 +1099,13 @@ def main(argv: Sequence[str] | None = None) -> DatasetGenerationResult:
     print(f"  Scripts: {skill['scripts'] or ['none']}")
     if skill.get("eval_prompt"):
         print(f"  Eval guidance: {skill['eval_prompt_source']}")
-    mode_parts = ["4-bucket" if args.full else "simple (1 test case)"]
+    if args.full:
+        if args.no_llm:
+            mode_parts = ["full bucket set (template; negative when authored in EVAL.md)"]
+        else:
+            mode_parts = ["full bucket set (4 cases via LLM)"]
+    else:
+        mode_parts = ["simple (1 test case)"]
     if args.refine:
         mode_parts.append("agent-refined")
     print(f"  Mode: {', '.join(mode_parts)}")

src/skillevaluator/tier3/harbor/report.py

@@ -705,7 +705,7 @@ def _passing_skill_suggestions(
     if num_trials < 4:
         suggestions.append(
             f"Expand evals.json with more test cases (currently {num_trials}). "
-            "Use 'skillevaluator create-eval-dataset --full' to generate a 4-bucket "
+            "Use 'skillevaluator create-eval-dataset --full' to generate the full bucket set "
             "strategy covering explicit, implicit, contextual, and negative cases."
         )