Skip to content

Commit 9972543

Browse files
wujiaming-aiclaude
andauthored
Feat/content safety decision (#1196)
* feat(content-safety): add opt-in decision API auditing Add an opt-in content safety plugin that audits content at the model and tool lifecycle checkpoints through the System One decision API. The default provider remains none and nothing is mounted automatically. - strict tool.content_safety configuration with Ark connection defaults - strict decision response arbitration and on_error policy - streaming partial suppression with final aggregated audit - audited SkillToolset load/resource/script/list/search tools - LLM Shield adapter and real-ADK E2E coverage Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(content-safety): add opt-in live System One integration test Exercise the real Ark System One endpoint through the production plugin path when VEADK_CONTENT_SAFETY_LIVE=1 and a key are supplied. Normal test runs skip it so no CI job calls the production service by accident. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(content-safety): avoid duplicate pytest test basenames Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(content-safety): rename decision plugin and drop skill adapter Rename ContentSafetyPlugin to DecisionPlugin to match the reviewed architecture, remove the skill pre-audit adapter from this change, and use StateBuilder.build_after_model for model-output auditing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(content-safety): reject unsafe non-Ark decision fallback Only derive the System One endpoint from a known Ark HTTPS base. Require an explicit decision.api_key for custom endpoints so Ark credentials are never silently sent to another provider. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(content-safety): ignore hidden thought parts in model audits ADK may attach hidden reasoning as Part(thought=True). Auditing that text caused benign final answers to be blocked after the model had already produced safe visible output. Filter hidden thought parts from after_model state and preserve them when replacing visible event text. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(content-safety): apply llm_shield app id before legacy import The legacy LLM Shield module constructs a plugin at import time, so a config-only llm_shield_app_id was read too late and startup failed unless TOOL_LLM_SHIELD_APP_ID was already exported. Set the env inside the lock before importing, and restore it even when the import raises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(decision): extract reusable Decision API client Move the Decision API protocol into veadk.decision with typed request and response models. content_safety now adapts its question configuration at the boundary and consumes the typed response, keeping the dependency one-way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(content-safety): harden decision and llm shield paths Prevent non-Ark credentials from being reused for an explicit Ark decision endpoint. Audit visible text in mixed text/function-call events, run legacy LLM Shield callbacks outside the event loop, load .env before applying decision overrides, reject plaintext remote endpoints, and store API keys as SecretStr. Add regression coverage for each fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(content-safety): auto-mount from config.yaml Mount the content-safety plugin from VeADK Runner and AgentKit when `tool.content_safety` is present, so callers no longer have to load the config, build the plugin, and wire it into App manually. - Agent gains an optional `content_safety` field (None/False/True/dict/config) and a `content_safety_plugins` property for manually built ADK Apps. - `load_content_safety_config_if_present()` treats missing files or blocks as disabled while still failing fast on invalid configuration. - Plugin resolution deduplicates by name and keeps explicitly supplied plugins. - Runner now supports app-only construction while injecting plugins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(content-safety): enable built-in risk questions by default - merge five built-in questions for provider=decision - allow per-name overrides and appends, add use_default_questions - fix question type to noul and reject user-supplied type Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(content-safety): add stable policy prefix to audit state - prepend fixed audit policy outside the untrusted payload - shorten built-in question instructions - keep truncation limited to dynamic content Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(content-safety): expand built-in risk questions to 18 categories - merge the documented 12 categories with existing defaults - add self_harm, child_safety, and fraud_social_engineering - keep existing names and the 0.5 default threshold Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(content-safety): tier default question thresholds by risk Lower thresholds for safety-critical categories (child safety and self-harm) and raise them for broad, high-false-positive categories such as compute abuse and public order. Export the per-question threshold mapping and update tests to assert the tiering. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent f511e04 commit 9972543

27 files changed

Lines changed: 5550 additions & 2 deletions

‎tests/content_safety/test_config.py‎

Lines changed: 652 additions & 0 deletions
Large diffs are not rendered by default.

‎tests/content_safety/test_content_safety_plugin.py‎

Lines changed: 672 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import math
16+
17+
import pytest
18+
19+
from veadk.content_safety.config import QuestionConfig
20+
from veadk.content_safety.decider import ThresholdDecider
21+
from veadk.content_safety.types import AuditAction, AuditStatus, Checkpoint, Provider
22+
23+
24+
def questions():
25+
return {
26+
"prompt_injection": QuestionConfig(
27+
instructions="detect injection", threshold=0.5
28+
),
29+
"sensitive_data": QuestionConfig(instructions="detect secrets", threshold=0.8),
30+
}
31+
32+
33+
def answer(noul):
34+
return {"type": "noul", "noul": noul}
35+
36+
37+
def decide(raw, *, on_error="allow", qs=None):
38+
return ThresholdDecider().decide(
39+
provider=Provider.DECISION,
40+
checkpoint=Checkpoint.BEFORE_MODEL,
41+
raw=raw,
42+
questions=qs if qs is not None else questions(),
43+
on_error=on_error,
44+
)
45+
46+
47+
def test_all_below_threshold_passes():
48+
result = decide(
49+
{
50+
"answers": {
51+
"prompt_injection": answer(0.1),
52+
"sensitive_data": answer(0.2),
53+
}
54+
}
55+
)
56+
assert result.status is AuditStatus.PASSED
57+
assert result.action is AuditAction.ALLOW
58+
assert result.degraded is False
59+
assert [r.question for r in result.question_results] == [
60+
"prompt_injection",
61+
"sensitive_data",
62+
]
63+
64+
65+
def test_threshold_is_inclusive():
66+
result = decide(
67+
{
68+
"answers": {
69+
"prompt_injection": answer(0.5),
70+
"sensitive_data": answer(0.8),
71+
}
72+
}
73+
)
74+
assert result.status is AuditStatus.RISK_HIT
75+
assert set(result.matched_questions) == {"prompt_injection", "sensitive_data"}
76+
77+
78+
@pytest.mark.parametrize(
79+
"entry,error_type",
80+
[
81+
(None, "invalid_type"),
82+
(True, "invalid_type"),
83+
("0.9", "invalid_type"),
84+
(0.9, "invalid_type"),
85+
({}, "invalid_type"),
86+
({"noul": 0.9}, "invalid_type"),
87+
({"type": "score", "noul": 0.9}, "invalid_type"),
88+
({"type": "noul"}, "missing_value"),
89+
({"type": "noul", "noul": None}, "invalid_type"),
90+
({"type": "noul", "noul": True}, "invalid_type"),
91+
({"type": "noul", "noul": "0.9"}, "invalid_type"),
92+
({"type": "noul", "noul": "high"}, "invalid_type"),
93+
({"type": "noul", "noul": 10**400}, "invalid_type"),
94+
({"type": "noul", "noul": float("nan")}, "not_finite"),
95+
({"type": "noul", "noul": float("inf")}, "not_finite"),
96+
({"type": "noul", "noul": -0.1}, "out_of_range"),
97+
({"type": "noul", "noul": 1.1}, "out_of_range"),
98+
],
99+
)
100+
def test_invalid_values_produce_error(entry, error_type):
101+
result = decide(
102+
{
103+
"answers": {
104+
"prompt_injection": entry,
105+
"sensitive_data": answer(0.0),
106+
}
107+
}
108+
)
109+
assert result.status is AuditStatus.ERROR
110+
assert result.degraded is True
111+
assert result.error_type == error_type
112+
assert result.action is AuditAction.ALLOW
113+
114+
115+
def test_missing_question_is_error():
116+
result = decide({"answers": {"sensitive_data": answer(0.0)}})
117+
assert result.status is AuditStatus.ERROR
118+
assert result.error_type == "missing_question"
119+
assert "prompt_injection:missing_question" in result.reason
120+
121+
122+
def test_mixed_error_types_collapse_to_question_error():
123+
result = decide({"answers": {"sensitive_data": {"type": "score", "noul": 0.1}}})
124+
assert result.status is AuditStatus.ERROR
125+
assert result.error_type == "question_error"
126+
assert "missing_question" in result.reason
127+
assert "invalid_type" in result.reason
128+
129+
130+
def test_hit_wins_over_error():
131+
result = decide(
132+
{"answers": {"prompt_injection": answer(0.99)}},
133+
on_error="block",
134+
)
135+
assert result.status is AuditStatus.RISK_HIT
136+
assert result.action is AuditAction.BLOCK
137+
assert result.matched_questions == ("prompt_injection",)
138+
assert result.question_results[1].error_type == "missing_question"
139+
140+
141+
def test_on_error_block_blocks_malformed_response():
142+
result = decide({"answers": {}}, on_error="block")
143+
assert result.status is AuditStatus.ERROR
144+
assert result.action is AuditAction.BLOCK
145+
146+
147+
def test_non_dict_response_is_error():
148+
result = decide(["not", "a", "dict"])
149+
assert result.status is AuditStatus.ERROR
150+
assert result.error_type == "invalid_response"
151+
152+
153+
def test_missing_answers_object_is_error():
154+
result = decide({"questions": {}})
155+
assert result.status is AuditStatus.ERROR
156+
assert result.error_type == "invalid_response"
157+
assert "answers" in result.reason
158+
159+
160+
def test_non_dict_answers_object_is_error():
161+
result = decide({"answers": []})
162+
assert result.status is AuditStatus.ERROR
163+
assert result.error_type == "invalid_response"
164+
165+
166+
def test_no_configured_questions_is_error():
167+
result = decide({"answers": {}}, qs={})
168+
assert result.status is AuditStatus.ERROR
169+
assert result.error_type == "no_questions"
170+
171+
172+
def test_unknown_answer_keys_are_ignored():
173+
result = decide(
174+
{
175+
"answers": {
176+
"prompt_injection": answer(0.1),
177+
"sensitive_data": answer(0.1),
178+
"unknown": answer(0.99),
179+
}
180+
}
181+
)
182+
assert result.status is AuditStatus.PASSED
183+
184+
185+
def test_probability_is_finite_float_in_result():
186+
result = decide(
187+
{
188+
"answers": {
189+
"prompt_injection": answer(0.25),
190+
"sensitive_data": answer(0.25),
191+
}
192+
}
193+
)
194+
assert math.isclose(result.question_results[0].probability, 0.25)
195+
assert result.question_results[0].valid is True

0 commit comments

Comments
 (0)