https://livekit.io logo
Join Slack
Powered by
How can I detect language automatically and switch...
# ask-ai
l
How can I detect language automatically and switch the LLM to give outputs in that language on the fly without much delay?
t
Thanks for reaching out, Ritesh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
You're asking how to auto-detect the input language and switch the LLM to respond in that language on the fly with minimal delay — here’s a concise, practical approach you can implement. 1) Use an STT that returns detected language • Choose a realtime/multilingual STT (examples: Deepgram Nova with 
language="multi"
, AssemblyAI Universal-Streaming-Multilingual, or ElevenLabs Scribe V2). These provide transcripts plus a detected language code per segment. 2) Pass the detected language to the LLM as an instruction • After you receive the transcription event, extract 
detected_language
(e.g., "es", "fr"). • Construct a short directive and include the user text, for example: ◦ Prompt: "Respond in {detected_language}: {user_transcript}" • Use a multilingual LLM (e.g., GPT-4.1-mini) that generates reliably in the requested language. 3) Keep prompts minimal to reduce latency • Only include necessary context and the language directive to keep model processing quick. • If you need a bit of context (system persona or short convo history), keep it compact. 4) Optionally switch TTS voice for playback • If you use TTS, pick a voice that matches the detected language and synthesize the LLM reply with it. 5) Pipeline example (minimal-latency flow) 1. STT (language="multi") streams audio → returns text + 
language
. 2. Immediately build short prompt: 
Respond in {language}: {text}
. 3. Send prompt to low-latency multilingual LLM. 4. Send LLM output to TTS (select voice by language) and play. 6) Latency tips • Use realtime-optimized STT and low-latency LLMs (e.g., Deepgram Nova, GPT-4.1-mini). • Reduce prompt size and avoid sending large history each turn. • Stream results where supported (receive partial transcriptions and incremental LLM responses). • If needed, use fast TTS voices or stream TTS audio. If you want, I can: • Show a small code snippet for your preferred runtime (Python/Node) using a specific STT + LLM pair, or • Recommend exact model choices and prompt templates tailored to your target languages and latency budget. Which would you prefer? Sources: Deepgram STT | LiveKit docs | AssemblyAI STT | LiveKit Docs | ElevenLabs STT | LiveKit docs | Live Conversations with AI using ChatGPT and WebRTC | Live Conversations with AI using ChatGPT and WebRTC
l
I use deepgram for stt google for tts and openai gpt 4o mini as llm. Give me a short config as per this
t
Below is a short, focused config + minimal handler logic for Deepgram STT (auto-detect), OpenAI GPT-4o-mini LLM, and Google TTS. It shows extracting the detected language, prompting the LLM to respond in that language, and sending the reply to Google TTS. Python-style pseudocode (keep keys/secrets out of source):
Copy code
# Configure services
stt = DeepgramSTT(model="nova-3", language="multi")   # auto-detect language
llm = OpenAIGPT(model="gpt-4o-mini")                 # multilingual LLM
tts = GoogleTTS(credentials=GOOGLE_CREDS)            # select voice by language

# Minimal event handler
async def on_transcription(event):
    text = event.text                    # transcription text
    lang = event.language                # detected language code, e.g. "es", "fr"

    # Short directive to LLM to minimize latency
    prompt = f"Respond in {lang}: {text}"

    # Call the LLM (use streaming if available)
    llm_resp = await llm.chat(prompt=prompt, max_tokens=300)

    # Synthesize with Google TTS using a voice appropriate for `lang`
    voice = voice_by_language(lang)      # map language -> Google voice
    audio = await tts.synthesize(text=llm_resp, voice=voice)

    # Play/send audio to user
    await send_audio_to_client(audio)

# Helper: simple language -> voice mapping
def voice_by_language(lang):
    return {
      "en": "en-US-Wavenet-D",
      "es": "es-ES-Wavenet-B",
      "fr": "fr-FR-Wavenet-B",
      # add others as needed
    }.get(lang, "en-US-Wavenet-D")
Key notes to keep latency low: • Keep the LLM prompt minimal (only language directive + transcription). • Use streaming for STT and LLM where available. • Pick fast voices in Google TTS and synthesize/play as soon as LLM returns. • Limit context/history per turn unless required. Would you like a Node.js snippet or a version with streaming calls included?
l
This is inside agent or the entry point function?
t
Place that code at your entry point where you wire up the AgentSession/pipeline — not inside the agent's decision logic. • Entry point: instantiate STT, LLM, TTS, and register the transcription event handler (the minimal event loop that receives transcriptions, extracts 
language
, prompts the LLM, and calls TTS). This keeps I/O, streaming, and service orchestration centralized. • Agent logic: keep short, focused reasoning/policy inside the agent (e.g., building system prompts, maintaining convo state, deciding actions). The agent should expose a simple function like 
agent.generate_response(text, lang)
which the entry-point handler calls. Minimal structure: 1. App/entry-point: ◦ Configure Deepgram STT (language="multi"), OpenAI LLM, Google TTS. ◦ Register 
on_transcription
to receive STT events. ◦ Call 
agent.generate_response(text, lang)
and then synthesize/play audio. 1. Agent module: ◦ Implements `generate_response(text, lang)`: ▪︎ Builds concise prompt (e.g., 
Respond in {lang}: {text}
). ▪︎ Calls LLM (streaming if available) and returns text. This separation keeps latency-sensitive I/O and streaming in the entry point and keeps the agent focused on generating the response. Would you like a short Node.js or Python example showing this split?
l
There is no event named on transcription but there is a on_user_turn_completed. The issue with this is the params are self , chat_ctx and new_message so there is no input field
t
You're right to point out 
on_user_turn_completed
— use 
new_message.text_content()
as the user input (the STT transcription). Here's a short, focused Python example showing exactly where to detect language and set the LLM/TTS flow inside 
on_user_turn_completed
.
Copy code
from livekit.agents import ChatContext, ChatMessage, voice
from langdetect import detect  # or use the STT's detected language if available

class MyAgent(voice.Agent):
    async def on_user_turn_completed(self, turn_ctx: ChatContext, new_message: ChatMessage) -> None:
        user_text = new_message.text_content()
        if not user_text:
            return

        # 1) Detect language (prefer STT-provided language if present)
        detected_lang = getattr(new_message, "language", None) or detect(user_text)  # e.g., "es", "fr"

        # 2) Short directive for LLM to respond in the same language
        # Keep the prompt minimal to reduce latency
        prompt = f"Respond in {detected_lang}: {user_text}"

        # 3) Call your LLM (OpenAI GPT-4o-mini) via your llm client
        llm_resp = await self.llm.chat(prompt=prompt, max_tokens=300)  # adapt to your client

        # 4) Synthesize with Google TTS (choose voice by language)
        voice_name = voice_by_language(detected_lang)
        audio = await self.tts.synthesize(text=llm_resp, voice=voice_name)

        # 5) Add assistant message and/or play audio
        turn_ctx.add_message(role="assistant", content=llm_resp)
        await self.play_audio_to_user(audio)  # replace with your playback method

def voice_by_language(lang):
    return {
      "en": "en-US-Wavenet-D",
      "es": "es-ES-Wavenet-B",
      "fr": "fr-FR-Wavenet-B",
    }.get(lang, "en-US-Wavenet-D")
Notes: • If your STT (Deepgram) returns a 
language
field on the message, prefer that over 
langdetect
for accuracy and speed. • Keep the LLM prompt minimal ("Respond in {lang}: {text}") to minimize latency. • Ensure your TTS call selects a matching voice; LiveKit’s Google TTS can also be set to auto-decide language if configured. If you want, I can convert this to a Node.js snippet or show how to use streaming calls for lower latency. Which would you prefer? Sources: Pipeline nodes and hooks | LiveKit docs | Turn detection and interruptions | LiveKit Docs
l
Can I use stt_node to get the language param
t
Short answer: Not reliably — Deepgram can auto-detect language, but LiveKit’s 
stt_node
does not consistently expose that detected-language field to the agent. You can still get the language, but you must either read it from the STT provider output (by overriding 
stt_node
or calling Deepgram directly) or perform language detection on the transcript inside 
on_user_turn_completed
. Two concise options you can implement now: 1) Prefer STT-provided language (best accuracy) • Override 
stt_node
to inspect raw transcription events from Deepgram (or use Deepgram’s API directly) and capture the provider’s 
language
field. • Propagate that language into the turn (e.g., set 
new_message.language
or store on 
turn_ctx
) so your agent can use it in 
on_user_turn_completed
. 2) Fallback: detect from transcript in 
on_user_turn_completed
(easiest) • In 
on_user_turn_completed
, get the transcribed text with 
new_message.text_content()
. • Run a fast language detector (or prefer STT's detected language if present via 
getattr(new_message, "language", None)
). • Use the detected code to build a minimal prompt: 
Respond in {lang}: {text}
and pick a matching Google TTS voice. Minimal snippet (inside your agent) showing the fallback pattern:
Copy code
user_text = new_message.text_content()
detected = getattr(new_message, "language", None) or fast_lang_detect(user_text)
prompt = f"Respond in {detected}: {user_text}"
reply = await self.llm.chat(prompt=prompt, max_tokens=300)
await self.tts.synthesize(text=reply, voice=voice_by_language(detected))
turn_ctx.add_message(role="assistant", content=reply)
If you want, I can: • Provide the 
stt_node
override example that parses Deepgram events (Python), or • Give a Node.js version that calls Deepgram directly and attaches language to the turn. Which would you like? Sources: Deepgram STT | LiveKit docs | Events and error handling | LiveKit docs | Speech-to-text (STT) models | LiveKit Docs
2 Views
Open in Slack
PreviousNext
Powered by