Skip to content

fix(media-use): voice ElevenLabs with eleven_v4 and native word times… - #5702

Open
kumanday wants to merge 1 commit into
heygen-com:mainfrom
kumanday:fix/media-use-elevenlabs-v4-timestamps
Open

kumanday wants to merge 1 commit into
heygen-com:mainfrom
kumanday:fix/media-use-elevenlabs-v4-timestamps

Conversation

@kumanday

Copy link
Copy Markdown

What

The media-use ElevenLabs TTS adapter now calls POST /v1/text-to-speech/{voice_id}/with-timestamps directly:

  • Model: eleven_v4 by default, overridable with --tts-model, tts_model in the request, or $ELEVENLABS_MODEL_ID.
  • Word timings: taken from the character alignment in the same response, so the Whisper pass no longer runs for ElevenLabs lines.
  • Python: ElevenLabs TTS no longer needs Python or pip install elevenlabs.

Why

skills/media-use/audio/scripts/lib/tts.mjs hard-coded model_id="eleven_multilingual_v2" in the Python helper, with no option to change it. Every workflow that voices through the shared audio engine therefore got the older model: faceless-explainer, product-launch-video, pr-to-video and others.

The provider table in references/tts.md listed ElevenLabs as having no word timestamps. Because of that, every ElevenLabs line was transcribed again with Whisper, which costs time and gives estimated timings. /with-timestamps returns exact character timings with the audio.

Related work

How

  • synthesizeElevenLabs: a new function in lib/tts.mjs, following the synthesizeHeygen pattern: injectable fetch and transcodeToWav, never throws, and returns { ok, words, error } with the reason on failure.
    • The request sends language_code from the line's lang. If a model rejects it with a 400 that mentions language, the call is retried once without it.
    • mp3 bytes are transcoded to wav as before, or written directly for .mp3 output.
  • elevenlabsWords: turns the alignment into [{text,start,end}] and skips bracketed audio tags ([warm], [slightly wry]), so tags never reach captions.
  • Provider detection: elevenlabsAvailable() now checks only for $ELEVENLABS_API_KEY, since no Python import is involved.
  • Docs (references/tts.md):
    • ElevenLabs now shows word timestamps in the provider table.
    • A short "ElevenLabs narration" section covers model, language and delivery.
    • The docs-site narrator section keeps its pinned eleven_multilingual_v2 voice. It now says the pin applies only to docs-site films. Switching that house voice is a separate call for maintainers.
  • Other docs: references/tts-to-captions.md and references/requirements.md reflect the new path.
  • skills-manifest.json: regenerated.

Test plan

  • Unit tests added/updated: lib/tts.test.mjs covers
    • alignment to words, with tags removed;
    • the request URL, body, default model_id and xi-api-key header;
    • explicit model override and the language_code retry;
    • .mp3 output into a new directory;
    • missing-key and HTTP-error messages.
  • node --test over skills/media-use/**/*.test.mjs, skills/faceless-explainer/**/*.test.mjs and skills/product-launch-video/**/*.test.mjs with the packages/cli/scripts/test-home.mjs import: 366 pass.
  • bun packages/cli/scripts/gen-skills-manifest.ts --check, bun scripts/lint-skills.ts, oxfmt --check and oxlint are clean on the changed files.
  • Manual testing performed: one live call through synthesizeOne({ provider: "elevenlabs", lang: "es", text: "[curious] ¿Y si el grafo aprende solo?" }) returned a 2.08 s, 44.1 kHz mono wav and six words with timings. The [curious] tag was excluded.
  • Documentation updated
  • Comments follow CONTRIBUTING.md "Comments"

…tamps

The ElevenLabs adapter hard-coded eleven_multilingual_v2 through the Python
SDK and then ran a Whisper pass for word timings. It now calls
/v1/text-to-speech/{voice}/with-timestamps directly, defaults to eleven_v4
(override with --tts-model, tts_model or ELEVENLABS_MODEL_ID), sends the
line's language as language_code, and builds voices[].words from the
character alignment with audio tags left out. ElevenLabs TTS no longer needs
Python or the elevenlabs package.
@kumanday
kumanday force-pushed the fix/media-use-elevenlabs-v4-timestamps branch from 4f1d5a2 to fa123fa Compare October 16, 2026 17:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant