Add skills eval setup with promptfoo

datajunction-clients/python/datajunction/skills/datajunction-repo.md

@@ -183,7 +183,7 @@ query: |
 # nodes/sources/transactions.yaml
 name: finance.transactions
 description: Raw transaction data from payment system
-type: source
+node_type: source
 catalog: prod_catalog
 schema_: finance
 table: transactions_table
@@ -227,7 +227,7 @@ mode: published
 # nodes/dimensions/user.yaml
 name: finance.user
 description: User dimension with attributes
-type: dimension
+node_type: dimension
 query: |
   SELECT
     user_id,
@@ -281,7 +281,7 @@ mode: published
 # nodes/metrics/revenue.yaml
 name: finance.total_revenue
 description: Total revenue from completed transactions
-type: metric
+node_type: metric
 query: |
   SELECT
     SUM(
@@ -319,7 +319,7 @@ mode: published
 # nodes/transforms/clean_transactions.yaml
 name: finance.clean_transactions
 description: Cleaned transaction data with standardized status
-type: transform
+node_type: transform
 primary_key:
   - transaction_id
 query: |
@@ -598,7 +598,7 @@ git checkout feature-add-churn-metric
 cat > nodes/metrics/churn_rate.yaml <<'EOF'
 name: finance.churn_rate
 description: Monthly user churn rate
-type: metric
+node_type: metric
 query: |
   SELECT
     CAST(SUM(CASE WHEN churned = true THEN 1 ELSE 0 END) AS DOUBLE) /

datajunction-clients/python/skill_evals/.env.example

@@ -0,0 +1,16 @@
+# Copy to .env and fill in (real .env is gitignored). The harness reads everything
+# from env — nothing about the provider is hardcoded.
+#
+# Bring your own key (default — uses the public OpenAI API):
+OPENAI_API_KEY=sk-...
+SKILL_EVAL_MODEL=gpt-4o
+#
+# OpenAI-compatible gateway/proxy instead of api.openai.com: point the base URL at it
+# and pick a model it serves. (If your org runs a billing proxy, its specifics go in a
+# local note, not here.)
+#   OPENAI_BASE_URL=http://your-openai-compatible-endpoint/v1
+#   OPENAI_API_KEY=...                 # whatever the endpoint expects
+#   SKILL_EVAL_MODEL=claude-sonnet-4-6 # any model the endpoint serves
+#
+# The python assertions need pyyaml. Point promptfoo at a Python that has it:
+#   PROMPTFOO_PYTHON=/path/to/a/venv/bin/python   # a venv with `pip install pyyaml`

datajunction-clients/python/skill_evals/.gitignore

@@ -0,0 +1,4 @@
+# generated by run.sh (local model override)
+promptfooconfig.local.yaml
+__pycache__/
+.env

datajunction-clients/python/skill_evals/README.md

@@ -0,0 +1,69 @@
+# Skill evals (behavioral)
+
+Behavioral evals for the bundled DJ Claude skills: load a `SKILL.md` into a model's
+context, give it a realistic request, and grade what it produces.
+
+**This is a local / on-demand tool, not CI.** It makes real LLM calls, so it isn't
+wired into GitHub Actions (public runners have no provider key). The free, every-PR
+safety net is the *programmatic* tier —
+`tests/test_skill_examples.py`, which validates the example YAML in the skills against
+the deployment schema with no model. This dir is the deliberate, occasional quality
+check on top of that.
+
+## What it checks
+
+Cases fall in two groups. **Authoring** cases (single turn) ask for a node and grade the
+YAML. **Decomposition** cases (two turns) test the semantic-model skill's "propose, don't
+produce" workflow: turn 1 should return a *structured decomposition proposal* (no YAML),
+turn 2 produces the deployment YAML after a follow-up. `provider.py` drives both turns in
+one test row so there's no fragile cross-row ordering.
+
+The semantic-modeling traps under test: ratio → named base metrics + a derived metric
+(not one inlined blob); filters via `CASE WHEN`, not a `WHERE`; slice-by-dimension via a
+dimension link, not a baked-in `JOIN`; the reusability rule holding even when the user
+asks for "just one metric"; mixed grains split into separate transforms; business-
+meaningful naming.
+
+Each case asserts with some mix of:
+
+- **programmatic, no LLM** — `assert_node.py` (single node) / `assert_deployment.py`
+  (multi-node): nodes parse, use `node_type` (not legacy `type:`), have required fields,
+  and satisfy case-specific shape (min nodes per type, a derived ratio metric exists, a
+  dimension link exists, no JOIN baked into a query, a query matches/avoids a pattern).
+  Rules are shared in `node_rules.py` so the two asserts can't drift;
+- **llm-rubric** (LLM judge) — the modeling judgment: did it decompose correctly, resist
+  the shortcut, name things meaningfully.
+
+## Running it
+
+The provider endpoint comes from env — nothing about it is hardcoded. Copy
+`.env.example` to `.env` (gitignored) and fill it in, then run `./run.sh`.
+
+**Bring your own key (default — public OpenAI API):**
+
+```bash
+export OPENAI_API_KEY="sk-..."                  # your OpenAI key
+./run.sh                                        # defaults to gpt-4o
+npx promptfoo@latest view                       # browse results
+```
+
+**Any OpenAI-compatible endpoint (e.g. a gateway/proxy):** point the base URL at it and
+pick a model it serves. `run.sh` generates a gitignored `promptfooconfig.local.yaml`
+with `SKILL_EVAL_MODEL` (the committed config stays `gpt-4o`, so BYO-key runs and
+gateway runs don't fight over one line):
+
+```bash
+export OPENAI_BASE_URL="http://your-endpoint/v1"
+export OPENAI_API_KEY="..."                     # whatever the endpoint expects
+SKILL_EVAL_MODEL=claude-sonnet-4-6 ./run.sh     # any model the endpoint serves
+```
+
+## Files
+
+- `promptfooconfig.yaml` — providers, prompts, golden cases + assertions.
+- `skill_prompt.py` — prompt function: injects the SKILL.md(s) as system + the request.
+  `vars.skill` is a comma-separated string (multiple skills compose).
+- `provider.py` — two-turn OpenAI-compatible client; a `followup` var triggers turn 2.
+- `node_rules.py` — shared node-spec rules mirroring the server deployment schema.
+- `assert_node.py` — structural check on a single produced node.
+- `assert_deployment.py` — structural check on a multi-node decomposition / deployment.

datajunction-clients/python/skill_evals/assert_deployment.py

@@ -0,0 +1,86 @@
+"""promptfoo python assertion for multi-node decomposition outputs (the turn-2 YAML of
+a two-turn case, or any answer that emits several nodes / a deployment doc).
+
+It validates every node structurally (via ``node_rules``) and then checks the
+case-specific modeling shape from test vars:
+  - ``min_nodes_by_type``     — mapping like ``{metric: 3, transform: 1}``: at least
+                                this many nodes of each type must be present;
+  - ``require_derived_ratio`` — at least one metric must be a derived ratio (an
+                                expression composing other metrics, no raw aggregate);
+  - ``require_dimension_link``— at least one node must declare ``dimension_links``;
+  - ``forbid_join``           — no metric/transform query may contain a ``JOIN`` (joins
+                                belong in dimension links, not baked into queries).
+
+If the output contains the turn-2 marker, only the part after it (the YAML turn) is
+validated — the proposal turn is graded by the llm-rubric, not here.
+"""
+
+import re
+
+import node_rules
+
+_JOIN = re.compile(r"\bJOIN\b", re.IGNORECASE)
+
+
+def get_assert(output, context):
+    variables = context.get("vars", {})
+
+    yaml_part = output
+    if node_rules.TURN2_MARKER in output:
+        yaml_part = output.split(node_rules.TURN2_MARKER, 1)[1]
+
+    nodes = [
+        n
+        for n in node_rules.parse_nodes(yaml_part)
+        if not node_rules.is_reference_stub(n)
+    ]
+    if not nodes:
+        return {
+            "pass": False,
+            "score": 0,
+            "reason": "no authored nodes found in deployment output",
+        }
+
+    problems: list[str] = []
+    for node in nodes:
+        problems.extend(node_rules.validate_node(node))
+
+    counts: dict[str, int] = {}
+    for node in nodes:
+        counts[node.get("node_type")] = counts.get(node.get("node_type"), 0) + 1
+
+    for node_type, minimum in (variables.get("min_nodes_by_type") or {}).items():
+        if counts.get(node_type, 0) < minimum:
+            problems.append(
+                f"expected >= {minimum} {node_type} node(s), found {counts.get(node_type, 0)}",
+            )
+
+    if variables.get("require_derived_ratio") and not any(
+        node_rules.is_derived_metric(n) for n in nodes
+    ):
+        problems.append(
+            "no derived ratio metric (expected a metric composing other metrics by name, "
+            "not a single query with the whole ratio inlined)",
+        )
+
+    if variables.get("require_dimension_link") and not any(
+        n.get("dimension_links") for n in nodes
+    ):
+        problems.append(
+            "no node declares `dimension_links` (the join should be a dim link)",
+        )
+
+    if variables.get("forbid_join"):
+        for node in nodes:
+            if node.get("node_type") in ("metric", "transform") and _JOIN.search(
+                str(node.get("query") or ""),
+            ):
+                problems.append(
+                    f"{node_rules.node_label(node)}: query contains a JOIN — joins belong "
+                    f"in dimension links, not baked into the query",
+                )
+
+    summary = ", ".join(f"{v}×{k}" for k, v in sorted(counts.items())) or "0 nodes"
+    if problems:
+        return {"pass": False, "score": 0, "reason": "; ".join(problems)}
+    return {"pass": True, "score": 1, "reason": f"valid deployment ({summary})"}

datajunction-clients/python/skill_evals/assert_node.py

@@ -0,0 +1,49 @@
+"""promptfoo python assertion: the model produced a single node conforming to the
+server deployment schema (``datajunction_server.models.deployment``). For multi-node
+decomposition outputs use ``assert_deployment.py``.
+
+Rules live in ``node_rules`` (shared with the deployment assert). Per-case expectations
+come from test vars:
+  - ``expected_node_type``  — the node_type the case asked for;
+  - ``require_in_query``    — regex the node's ``query`` MUST match (e.g. ``CASE\\s+WHEN``);
+  - ``forbid_in_query``     — regex the node's ``query`` must NOT match (e.g. a ``WHERE``).
+"""
+
+import re
+
+import node_rules
+
+
+def get_assert(output, context):
+    variables = context.get("vars", {})
+    expected = variables.get("expected_node_type")
+
+    nodes = node_rules.parse_nodes(output)
+    if not nodes:
+        return {"pass": False, "score": 0, "reason": "no YAML node found in output"}
+    if len(nodes) > 1:
+        return {
+            "pass": False,
+            "score": 0,
+            "reason": f"expected a single node, found {len(nodes)} (use a deployment case)",
+        }
+
+    data = nodes[0]
+    problems = node_rules.validate_node(data, expected)
+
+    query = str(data.get("query") or "")
+    require = variables.get("require_in_query")
+    if require and not re.search(require, query):
+        problems.append(f"query does not match required pattern {require!r}")
+    forbid = variables.get("forbid_in_query")
+    if forbid and re.search(forbid, query):
+        problems.append(f"query matches forbidden pattern {forbid!r}")
+
+    if problems:
+        return {"pass": False, "score": 0, "reason": "; ".join(problems)}
+    node_type = data.get("node_type")
+    return {
+        "pass": True,
+        "score": 1,
+        "reason": f"valid {node_type} node per deployment schema",
+    }