Skip to content

fix(llm): add generic opt-in tool-call text policy for all text LLM providers - #5369

Draft
martinstiago wants to merge 2 commits into
pipecat-ai:mainfrom
martinstiago:fix/openai-post-tool-call-text-policy
Draft

fix(llm): add generic opt-in tool-call text policy for all text LLM providers#5369
martinstiago wants to merge 2 commits into
pipecat-ai:mainfrom
martinstiago:fix/openai-post-tool-call-text-policy

Conversation

@martinstiago

@martinstiago martinstiago commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds an opt-in ToolCallTextPolicy setting (PRESERVE / SUPPRESS_AFTER_TOOL_CALL_DETECTED) that suppresses streamed text after a tool call is detected in the same response.
  • The setting lives on base LLMSettings, so every LLM service accepts it uniformly and it's runtime-updatable via LLMUpdateSettingsFrame.
  • Suppression is centralized in LLMService._push_llm_text: the base class owns a _tool_call_detected flag (reset per response via LLMFullResponseStartFrame in push_frame) and drops text when the policy is SUPPRESS_AFTER_TOOL_CALL_DETECTED. Providers call _note_tool_call_detected() at their tool-call detection site — one line per streaming loop.
  • Covers all text-bearing LLM services: OpenAI-compatible (20+ subclasses via BaseOpenAILLMService), OpenAI Responses (WebSocket + HTTP), Anthropic, Google (Gemini + Vertex), and AWS Bedrock Converse. Realtime speech-to-speech services set tool_call_text_policy=None (unsupported — they don't stream TTS-bound text).
  • Default is PRESERVE everywhere — non-breaking; no behavior changes unless users explicitly opt in.

Testing

  • uv run pytest tests/test_llm_service_tool_call_text.py tests/test_openai_tool_call_text.py tests/test_bedrock_tool_call_text.py tests/test_anthropic_tool_call_text.py tests/test_google_tool_call_text.py tests/test_openai_responses_tool_call_text.py
  • uv run ruff check src/pipecat/services/ tests/test_*tool_call_text.py
  • Validated end-to-end with real API calls: OpenAI chat completions, OpenAI Responses, and AWS Bedrock Converse (nova-lite + claude). Function calls execute with correct args in both policies; pre-tool-call text (narration, thinking) is preserved; post-tool-call text is suppressed when the policy is SUPPRESS_AFTER_TOOL_CALL_DETECTED; suppression state resets between responses.

Documentation

@martinstiago martinstiago changed the title fix(llm): add opt-in tool-call text policy fix(llm): add generic opt-in tool-call text policy for all text LLM providers Aug 21, 2026
@martinstiago
martinstiago force-pushed the fix/openai-post-tool-call-text-policy branch from 4acb9a6 to 0337769 Compare August 21, 2026 21:49
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
src/pipecat/services/anthropic/llm.py 71.22% <100.00%> (+1.55%) ⬆️
src/pipecat/services/aws/llm.py 70.43% <100.00%> (+0.95%) ⬆️
src/pipecat/services/aws/nova_sonic/llm.py 29.52% <ø> (ø)
src/pipecat/services/google/gemini_live/llm.py 29.30% <ø> (ø)
src/pipecat/services/google/llm.py 72.34% <100.00%> (+0.08%) ⬆️
src/pipecat/services/inworld/realtime/llm.py 30.91% <ø> (ø)
src/pipecat/services/llm_service.py 90.14% <100.00%> (+0.10%) ⬆️
src/pipecat/services/openai/base_llm.py 86.13% <100.00%> (+14.49%) ⬆️
src/pipecat/services/openai/realtime/llm.py 54.88% <ø> (ø)
src/pipecat/services/openai/responses/llm.py 91.05% <100.00%> (+0.39%) ⬆️
... and 3 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…_CALL_DETECTED

Renames the enum member and value per review feedback for clarity.
@martinstiago

Copy link
Copy Markdown
Contributor Author

Renamed the enum member and value to SUPPRESS_AFTER_TOOL_CALL_DETECTED / suppress_after_tool_call_detected per Weitian's naming feedback — clearer that suppression kicks in once a tool call is detected. Docs PR updated to match.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant