Hey everyone, I’m building a multilingual voice ag...
# agents
m
Hey everyone, I’m building a multilingual voice agent using LiveKit Agents together with ElevenLabs STT/TTS and GPT-4o, and I’ve run into a problem around reliable language handling that I’d appreciate some guidance on. Functionally, things are working well. ElevenLabs STT accurately transcribes many different languages, and GPT-4o responds in the correct language based on the user’s input. ElevenLabs TTS can also synthesize multilingual text reasonably well. So from a conversational standpoint, the system already works. The issue is with language metadata and TTS voice routing, not transcription. The STT transcript is correct, but the language code coming through the current ElevenLabs Scribe v1 integration (via
livekit-plugins-elevenlabs
1.3.3) is always reported as
"en"
. That means I don’t have a reliable signal for what language the user actually spoke during the last utterance. This becomes a problem because I want to do things like: • Track which language a session is in • Choose the correct TTS voice for that language • Switch dialects/accents per language where it improves the experience I know STTv2 / Scribe v2 Realtime recently landed in a PR, and I’m hoping it will expose accurate language detection in the WebSocket stream, but I’m not sure what to expect or whether it’s available yet on PyPI. The bigger challenge is on the TTS side. ElevenLabs STT (especially with Scribe v2) can auto-detect a very wide range of languages, but TTS voices don’t cover nearly as many. Even the best Turbo/Flash v2.5 voices support around 30–32 languages, and a single voice often doesn’t sound natural across all of them. So even if I correctly detect the spoken language, I still need a way to route the LLM’s output to the appropriate voice ID that best matches the target language, accent, and dialect. Without reliable language metadata from STT, I can’t do this cleanly. Right now I’m considering adding lightweight language detection on the text itself (e.g., before passing to TTS), but I’m trying to avoid unnecessary latency and complexity if the STTv2 integration is meant to solve this. My main questions for the community are: 1. With STTv2 / Scribe v2 Realtime, will LiveKit Agents expose accurate per-utterance language codes so I can reliably determine the last user language? 2. What’s the recommended way to route TTS output when one voice doesn’t cover all languages well and I need to switch between multiple voice IDs to get better coverage and accents? 3. Is there an existing pattern for mapping detected STT language → best TTS voice profile inside the LiveKit agent loop?
many times to switch settings you use
update_options()
🙌 1