Skip to content

fix(gemini): migrate to google.golang.org/genai SDK - #127

Merged
devops-bda merged 1 commit into
mainfrom
fix/gemini-genai-sdk
Oct 8, 2026
Merged

devops-bda merged 1 commit into
mainfrom
fix/gemini-genai-sdk

Conversation

@nirmal-bluefunda

Copy link
Copy Markdown
Contributor

Problem

The Gemini provider used github.com/google/generative-ai-go, which Google has deprecated. It can't parse current Gemini 3.x responses:

  • Every stream ends in invalid character ']' looking for beginning of value, even after a complete answer, so callers treat good answers as errors.
  • Function-calling turns come back as an unknown finish reason (FinishReason(10)) with no content.

Older models that worked with that SDK (e.g. gemini-2.5-flash) are retired for new users, so a model change can't fix it. In cai-llm-router this broke the Gemini model and the tool-retry-fallback (Groq → Gemini) recovery.

Changes

  • Provider rewritten on google.golang.org/genai (v1.73.0) in providers/gemini; the old SDK is removed from go.mod.
  • Gemini 3 thought signatures: new optional ToolCall.ThoughtSignature field (json:"thought_signature,omitempty"), set from the function-call part and replayed in history. Gemini 3 rejects follow-up tool-loop requests without it. Empty for every other provider.
  • Unique tool-call IDs and Index: Gemini often omits call IDs, and the old provider used the function name as the ID, so parallel calls to the same tool collided (cai-llm-router#294).
  • Tool history without declared tools is sent as text: Gemini rejects function-call history when the request has no tools (e.g. a tool-free retry).
  • Usage reported, with thinking tokens counted as completion tokens (previously missing).
  • Tool schemas passed as raw JSON Schema (parametersJsonSchema), so keywords aren't dropped.
  • The API's real error and HTTP status are surfaced through APIError.
  • Thought parts are skipped (never mixed into answer text). BaseURL and CustomHeaders are honoured.

No other provider changes. Only cai-llm-router depends on this module.

Tests

New providers/gemini/provider_test.go against a fake Gemini server (no key needed):

  • a stream ends cleanly, with usage;
  • tool calls keep the signature and get unique IDs and indexes;
  • thought text never leaks into the answer;
  • the follow-up request on the wire replays the signature, groups parallel results, resolves a missing tool name, and keeps the schema intact;
  • without tools, tool history is replayed as text;
  • API errors (404 with the server message) are reported;
  • Complete() works;
  • finish-reason mapping.

go vet ./... clean; all packages pass.

Live check: cai-llm-router run locally against this branch (go.mod replace) with the real API and gemini-3.5-flash-lite:

  • a plain chat answered cleanly;
  • a multi-step tool loop (4 model steps, 3 tool calls) completed with the signatures accepted.

🤖 Generated with Claude Code

The provider used github.com/google/generative-ai-go, which Google
deprecated and which cannot parse Gemini 3.x responses: every stream ended
in "invalid character ']'" (turning complete answers into errors), and
function-calling turns came back as an unknown finish reason with no text.

- Rewrite the Gemini provider on google.golang.org/genai.
- Carry Gemini 3 thought signatures on ToolCall.ThoughtSignature and
  replay them, so multi-step tool loops are accepted.
- Give each function call a unique ID and Index (parallel calls to the
  same tool used to collide on the function name).
- Without declared tools, replay tool history as text (Gemini rejects
  function-call history when no tools are declared).
- Report usage (thinking tokens count as completion), pass tool schemas
  through as raw JSON Schema, surface the API's real error and status,
  skip thought parts, honour BaseURL and CustomHeaders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nirmal-bluefunda
nirmal-bluefunda requested a review from a team as a code owner October 8, 2026 10:02
@devops-bda
devops-bda merged commit 46e15e7 into main Oct 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants