Skip to content

fix(media-use): voice ElevenLabs with eleven_v4 and native word times… #5702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kumanday wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(media-use): voice ElevenLabs with eleven_v4 and native word times… #5702

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion skills-manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
"files": 1
},
"media-use": {
"hash": "c4c3c723aa04bb7d",
"hash": "8da44b4cf2f6cfe4",
"files": 112
},
"motion-graphics": {
Expand Down
2 changes: 1 addition & 1 deletion skills/media-use/audio/references/requirements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Run `npx hyperframes auth status` to see what's configured and which engines a w
| Provider | Resolution order (first non-empty wins) | Local deps when used |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| **HeyGen** (TTS + BGM/SFX retrieval) | `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → `~/.heygen/credentials` (shared with heygen-cli; `$HEYGEN_CONFIG_DIR` overrides the dir; written by `hyperframes auth login`) | none (REST) |
| **ElevenLabs** (TTS fallback) | `$ELEVENLABS_API_KEY` | `pip install elevenlabs` |
| **ElevenLabs** (TTS fallback) | `$ELEVENLABS_API_KEY` | none (REST) |
| **Lyria** (BGM fallback) | `$GEMINI_API_KEY` → `$GOOGLE_API_KEY` | `pip install google-genai` |
| **Kokoro** (TTS, no key) | always — final voice fallback | `pip install kokoro-onnx soundfile` |
| **MusicGen** (BGM, no key) | always — final music fallback | `pip install transformers torch soundfile numpy` |
Expand Down
6 changes: 5 additions & 1 deletion skills/media-use/audio/references/tts-to-captions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@ node skills/media-use/audio/scripts/heygen-tts.mjs \

`narration.words.json` is already in the `[{ id, text, start, end }]` shape the captions pipeline consumes — no separate transcribe pass.

## Path B — Gemini / ElevenLabs / Kokoro (TTS → transcription)
ElevenLabs works the same way inside the shared audio engine: its adapter calls
`/with-timestamps` and fills `audio_meta.json` → `voices[].words` from the
character alignment, so no transcription runs (see [ElevenLabs narration](tts.md#elevenlabs-narration)).

## Path B — Gemini / Kokoro (TTS → transcription)

These adapters supply audio without word data. The shared audio engine runs
transcription automatically when timings are absent. For Gemini, use the
Expand Down
23 changes: 20 additions & 3 deletions skills/media-use/audio/references/tts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ re-voiced. If you cannot reach ElevenLabs, say so and stop rather than
substituting a different voice.

Use another voice only for a documented reason, and write the reason down.
This pin applies only to docs-site films. Elsewhere, the audio engine's
ElevenLabs adapter defaults to `eleven_v4` (see [ElevenLabs narration](#elevenlabs-narration)).

## Available routes

Expand All @@ -43,7 +45,7 @@ to HeyGen sign-in. Read the Gemini section for its credential requirement.
| Order | Provider | Env trigger | Voice IDs | Word timestamps | Audio format |
| ----- | ----------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------------------- | -------------------- |
| 1 | HeyGen (Starfish) | `$HEYGEN_ACCESS_TOKEN` / `$HEYGEN_API_KEY` / `~/.heygen/credentials` | UUIDs from `GET /v3/voices?engine=starfish`, or your cloned voice ids (`heygen-voice.mjs`) | **Yes** (`word_timestamps[]` in response) | mp3 → wav via ffmpeg |
| 2 | ElevenLabs | `$ELEVENLABS_API_KEY` | UUIDs from elevenlabs.io dashboard | No | mp3 → wav via ffmpeg |
| 2 | ElevenLabs | `$ELEVENLABS_API_KEY` | UUIDs from elevenlabs.io dashboard | **Yes** (`/with-timestamps`) | mp3 → wav via ffmpeg |
| 3 | Kokoro-82M | always (local fallback) | `am_michael`, `af_heart`, … (54 voices) | No | wav direct |

```bash
Expand Down Expand Up @@ -107,11 +109,26 @@ node skills/media-use/audio/scripts/heygen-voice.mjs delete <voice_id>
| Goal | Use |
| --------------------------------------------------------- | --------------------------------------------------------- |
| Best voice quality + word timestamps in one call | **HeyGen** |
| Drop-in cloud TTS, big voice catalog | **ElevenLabs** |
| Big voice catalog + word timestamps in one call | **ElevenLabs** |
| Offline, no API key, fast iteration | **Kokoro** |
| Directed delivery with Gemini prebuilt or custom voices | **Gemini** (explicit selection; transcription for timing) |
| Non-English multilingual with deterministic phonemization | **Kokoro** (`ef_dora`, `jf_alpha`, `zf_xiaobei`, …) |

## ElevenLabs narration

The shared audio engine calls `POST /v1/text-to-speech/{voice_id}/with-timestamps`
directly (no Python SDK) and builds `voices[].words` from the returned
character alignment, so no transcription pass runs.

- **Model:** `eleven_v4` by default. Override it with `--tts-model`,
`tts_model` in the request, or `$ELEVENLABS_MODEL_ID`.
- **Language:** the request's `lang` is sent as `language_code`. If a model
rejects it, the adapter retries once without it.
- **Delivery:** `eleven_v4` reads lowercase audio tags such as `[warm]` or
`[curious]` as delivery cues. Use at most one per line. Tags are kept out of
`words[]`, so captions never show them. `eleven_v4` ignores `speed`; pace
the script with sentence length and punctuation.

## Gemini narration

Use the shared audio engine, not `hyperframes tts`. Authenticate with either:
Expand Down Expand Up @@ -257,4 +274,4 @@ When `--words <path>` is passed to a HeyGen call, the file is written in the sam
]
```

For ElevenLabs / Kokoro, run `npx hyperframes transcribe narration.wav --model small.en` to get the same shape.
ElevenLabs returns the same shape through the audio engine. For Kokoro, run `npx hyperframes transcribe narration.wav --model small.en` to get it.
2 changes: 1 addition & 1 deletion skills/media-use/audio/scripts/audio.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ if (only.has("tts") && lines.length) {
anomalies.push(`line ${id}: TTS failed — omitted${error ? ` (${error})` : ""}`);
return null;
}
let wordArr = words; // heygen: native; else transcribe
let wordArr = words; // heygen/elevenlabs: native; else transcribe
if (!wordArr) wordArr = await transcribeWav({ wavRel: rel, lang, hyperframesDir });
const dur = ffprobeDuration(abs);
if (!isFinite(dur) || dur <= 0) {
Expand Down
4 changes: 2 additions & 2 deletions skills/media-use/audio/scripts/lib/python.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// python.mjs — resolve which Python 3 executable to spawn, per platform.
//
// The audio engine (tts.mjs, bgm.mjs) shells out to `python3` for ElevenLabs
// TTS and the local Lyria/MusicGen BGM paths. `python3` is the right name on
// The audio engine (bgm.mjs, gemini-auth.mjs) shells out to `python3` for the
// local Lyria/MusicGen BGM paths and Gemini service-account auth. `python3` is the right name on
// macOS/Linux, but on Windows the python.org installer only creates
// `python.exe` plus the `py` launcher — there is no `python3.exe` (only the
// Microsoft Store build adds one). So a bare `spawn("python3", …)` ENOENTs on a
Expand Down
6 changes: 3 additions & 3 deletions skills/media-use/audio/scripts/lib/tts.cleanup.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ if (["allocation", "write", "spawn-throw"].includes(mode)) {
assert.equal(fs.existsSync(wavAbs), false);
} else if (mode === "concurrent") {
const slowWav = join(project, "slow.wav");
const slow = request("elevenlabs", "slow", slowWav);
const slow = request("kokoro", "slow", slowWav);
try {
const deadline = Date.now() + 5000;
while (!fs.existsSync(slowWav + ".started")) {
Expand All @@ -93,7 +93,7 @@ if (["allocation", "write", "spawn-throw"].includes(mode)) {
assert.equal(fs.readFileSync(slowWav, "utf8"), "slow");
} else {
const result = await request();
const label = provider === "elevenlabs" ? "elevenlabs (python)" : "kokoro (npx hyperframes tts)";
const label = "kokoro (npx hyperframes tts)";
if (["nonzero", "enoent", "no-output"].includes(mode)) {
const error = mode === "no-output" ? label + " produced no wav file" :
label + " exited with status " + (mode === "enoent" ? -1 : 7);
Expand Down Expand Up @@ -149,7 +149,7 @@ function runCase(provider, mode) {
}
}

for (const provider of ["elevenlabs", "kokoro"]) {
for (const provider of ["kokoro"]) {
for (const mode of [
"success",
"nonzero",
Expand Down
132 changes: 89 additions & 43 deletions skills/media-use/audio/scripts/lib/tts.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ import { fetchMedia } from "../../../scripts/lib/media-fetch.mjs";
// Direct v3 REST (NOT `hyperframes tts`, which in the published build is
// Kokoro-only and silently ignores a HeyGen key). Returns word_timestamps
// in the same call, so no separate transcribe pass.
// 2. ElevenLabs — $ELEVENLABS_API_KEY + `pip install elevenlabs`. No
// word timings → caller chains transcribeWav().
// 2. ElevenLabs — $ELEVENLABS_API_KEY. Direct REST to /with-timestamps
// (default model eleven_v4); character alignment gives word timings in
// the same call, so no separate transcribe pass.
// 3. Kokoro-82M (local) — always available, via the published `hyperframes tts`
// CLI. No word timings → caller chains transcribeWav().
//
Expand All @@ -19,7 +20,6 @@ import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { heygenAuthHeaders, heygenCredential, heygenJSON, heygenMessage } from "./heygen.mjs";
import { pythonInvocation } from "./python.mjs";
import { synthesizeGemini } from "./gemini-tts.mjs";
import { geminiConfigured } from "./gemini-auth.mjs";

Expand All @@ -28,12 +28,7 @@ export function heygenAvailable() {
return heygenCredential() !== null;
}
export function elevenlabsAvailable() {
if (!process.env.ELEVENLABS_API_KEY) return false;
const { cmd, args } = pythonInvocation(["-c", "import elevenlabs"]);
const r = spawnSync(cmd, args, {
stdio: "ignore",
});
return r.status === 0;
return Boolean(process.env.ELEVENLABS_API_KEY);
}

// First available provider wins; an explicit choice is honored (and validated).
Expand Down Expand Up @@ -229,23 +224,91 @@ function transcodeToWav(bytes, destWav) {
return ff.status === 0 && existsSync(destWav);
}

const ELEVENLABS_PY = `
import os, sys
from elevenlabs.client import ElevenLabs
from elevenlabs import save
client = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])
text = open(sys.argv[1], encoding="utf-8").read()
audio = client.text_to_speech.convert(
text=text, voice_id=sys.argv[2],
model_id="eleven_multilingual_v2", output_format="mp3_44100_128",
)
save(audio, sys.argv[3])
`;
export const ELEVENLABS_DEFAULT_MODEL = "eleven_v4";
const ELEVENLABS_TTS_URL = "https://api.elevenlabs.io/v1/text-to-speech";

// Character alignment → [{text,start,end}]. Audio tags such as [warm] are voiced
// as delivery cues but are not words, so they are dropped from the timings.
export function elevenlabsWords(alignment) {
const chars = alignment?.characters ?? [];
const starts = alignment?.character_start_times_seconds ?? [];
const ends = alignment?.character_end_times_seconds ?? [];
const words = [];
let cur = null;
let inTag = false;
chars.forEach((ch, k) => {
if (ch === "[") inTag = true;
const gap = inTag || /\s/.test(ch);
if (ch === "]") inTag = false;
if (gap) {
if (cur) words.push(cur);
cur = null;
return;
}
if (!isFinite(starts[k]) || !isFinite(ends[k])) return;
if (cur) {
cur.text += ch;
cur.end = ends[k];
} else cur = { text: ch, start: starts[k], end: ends[k] };
});
if (cur) words.push(cur);
return words;
}

export async function synthesizeElevenLabs({ text, voiceId, lang, model, wavAbs }, deps = {}) {
const fetchImpl = deps.fetch ?? fetch;
const transcode = deps.transcodeToWav ?? transcodeToWav;
const key = process.env.ELEVENLABS_API_KEY;
if (!key) return { ok: false, words: null, error: "$ELEVENLABS_API_KEY is not set" };
const modelId = model || process.env.ELEVENLABS_MODEL_ID || ELEVENLABS_DEFAULT_MODEL;
const url = `${ELEVENLABS_TTS_URL}/${encodeURIComponent(voiceId)}/with-timestamps?output_format=mp3_44100_128`;
const post = (body) =>
fetchImpl(url, {
method: "POST",
headers: { "xi-api-key": key, "Content-Type": "application/json" },
body: JSON.stringify(body),
});
try {
const body = { text, model_id: modelId };
if (lang) body.language_code = lang;
let res = await post(body);
let detail = res.ok ? "" : await res.text();
// Some models reject language_code; the text alone still sets the language.
if (!res.ok && res.status === 400 && body.language_code && /language/i.test(detail)) {
delete body.language_code;
res = await post(body);
detail = res.ok ? "" : await res.text();
}
if (!res.ok) {
return {
ok: false,
words: null,
error: `ElevenLabs ${modelId} HTTP ${res.status}: ${detail.slice(0, 200)}`,
};
}
const payload = await res.json();
if (!payload.audio_base64) {
return { ok: false, words: null, error: "ElevenLabs /with-timestamps returned no audio" };
}
const bytes = Buffer.from(payload.audio_base64, "base64");
if (wavAbs.endsWith(".wav")) {
if (!transcode(bytes, wavAbs)) {
return { ok: false, words: null, error: "wav transcode failed (ffmpeg)" };
}
} else {
mkdirSync(dirname(wavAbs), { recursive: true });
writeFileSync(wavAbs, bytes);
}
return { ok: true, words: elevenlabsWords(payload.alignment ?? payload.normalized_alignment) };
} catch (e) {
return { ok: false, words: null, error: `ElevenLabs request failed: ${e?.message ?? e}` };
}
}

// ── synthesize one line ───────────────────────────────────────────────────────
// Writes wav at wavAbs. Returns { ok, words, error } — words is the raw
// [{text,start,end}] array for HeyGen (native), or null for ElevenLabs/Kokoro
// (caller must transcribeWav). Never throws; failures return { ok:false, error }
// [{text,start,end}] array for HeyGen and ElevenLabs (native), or null for
// Kokoro/Gemini (caller must transcribeWav). Never throws; failures return { ok:false, error }
// where `error` states WHY (so the caller can surface it, not a bare "TTS failed").
export async function synthesizeOne({
provider,
Expand All @@ -261,25 +324,8 @@ export async function synthesizeOne({
if (provider === "heygen") return synthesizeHeygen({ text, voiceId, lang, speed, wavAbs });
if (provider === "gemini")
return synthesizeGemini({ text, voiceId, model, style, speed, wavAbs });
if (provider === "elevenlabs") {
// The Python helper writes straight to wavAbs; unlike heygen (transcodeToWav)
// and kokoro (the `hyperframes tts` CLI), it does NOT create the parent dir,
// so on a fresh project (no assets/voice/ yet) the save fails and the line is
// silently dropped as "TTS failed - omitted". Create it first, like the other
// providers do. Guarded so a mkdir failure (EACCES/EROFS) returns
// { ok:false } like the rest of this branch rather than throwing (the
// function's contract is "never throws; failures return { ok:false }").
try {
mkdirSync(dirname(wavAbs), { recursive: true });
} catch {
return { ok: false, words: null };
}
return withTmpText(text, async (textPath) => {
const { cmd, args } = pythonInvocation(["-c", ELEVENLABS_PY, textPath, voiceId, wavAbs]);
const r = await spawnP(cmd, args, {});
return synthResult(r, wavAbs, "elevenlabs (python)");
});
}
if (provider === "elevenlabs")
return synthesizeElevenLabs({ text, voiceId, lang, model, wavAbs });
// kokoro — via the published CLI; --output is relative to the project dir.
const wavRel = relTo(hyperframesDir, wavAbs);
return withTmpText(text, async (textPath) => {
Expand Down Expand Up @@ -352,7 +398,7 @@ export async function synthesizeHeygen({ text, voiceId, lang, speed, wavAbs }, d
}
}

// ElevenLabs/Kokoro have no word timings — run Whisper over the wav. Returns the
// Kokoro/Gemini have no word timings — run Whisper over the wav. Returns the
// flat [{id,text,start,end}] word array, or null. Each call uses a throwaway
// --dir so parallel scenes don't collide on transcript.json.
export async function transcribeWav({ wavRel, lang = "en", hyperframesDir }) {
Expand Down
Loading