Hey everyone! :wave: I’m building a multilingual ...
# agents
m
Hey everyone! 👋 I’m building a multilingual voice agent using livekit-agents with the langchain.LLMAdapter + ElevenLabs TTS. The agent needs to dynamically switch TTS voice/language based on what language the LLM responds in (e.g., user speaks Spanish → LLM replies in Spanish → TTS should use Spanish voice). Current approach (polling): I’m polling the LangGraph state every 0.5s to detect new AI messages, run language detection, and call tts.update_options():
Copy code
async def monitor_agent_responses():
    """Monitor LLM responses and switch TTS language/voice accordingly"""
    last_message_id = None
    while True:
        try:
            await asyncio.sleep(0.5)  # Polling interval
            
            state = graph.get_state(config={"configurable": {"thread_id": thread_id}})
            if state and state.values.get('messages'):
                messages = state.values['messages']
                if messages:
                    last_msg = messages[-1]
                    
                    # Check if it's a new AI message
                    if (hasattr(last_msg, 'id') and last_msg.id != last_message_id and 
                        hasattr(last_msg, 'content') and last_msg.content and
                        not isinstance(last_msg, HumanMessage)):
                        
                        last_message_id = last_msg.id
                        text = last_msg.content
                        
                        # Detect language from AI response
                        if text and len(text.strip()) >= 3:
                            detected_results = detect(text, model='auto')  # fast_langdetect
                            
                            if detected_results and detected_results[0]['score'] >= 0.70:
                                lang_code = detected_results[0]['lang'].lower()
                                voice_id = LANGUAGE_VOICE_MAPPING.get(lang_code)
                                
                                if lang_code != current_language["code"]:
                                    <http://logger.info|logger.info>(f":earth_africa: Language switch: {current_language['code']} → {lang_code}")
                                    tts.update_options(language=lang_code, voice_id=voice_id)
                                    current_language["code"] = lang_code
        
        except Exception as e:
            logger.debug(f"Monitor error: {e}")
            await asyncio.sleep(1)

# Started as background task
monitor_task = asyncio.create_task(monitor_agent_responses())
The problem: The 0.5s polling interval creates a race condition — TTS often starts speaking before the monitor detects the language change, so the first sentence (or entire response) plays in the wrong voice/language. What I’ve tried: 1. Wrapping/subclassing elevenlabs.TTS to intercept .stream() — failed because the Agent framework didn’t call my wrapped methods. 2. Monkey-patching tts.stream/tts.synthesize on the instance — methods weren’t invoked. 3. class swizzling (changing the instance’s class at runtime) — failed with class assignment: ‘MultilingualTTS’ object layout differs from ‘TTS’ because the ElevenLabs plugin has a fixed C-level layout. 4. Faster polling (150ms, 50ms) — helps but doesn’t eliminate the race, and increases CPU load. My question: Is there a cleaner way to hook into the TTS pipeline before audio generation starts? Ideally something like: • A callback/event when Agent is about to synthesize text (so I can detect language and update TTS options first) • A way to wrap or extend SynthesizeStream that the framework will actually use • An event bus or signal when LLMAdapter yields a new chunk Or is polling the LangGraph state the only viable approach here? Setup: • livekit-agents with Agent, AgentSession, langchain.LLMAdapter • livekit-plugins-elevenlabs for TTS • LangGraph for conversation state • fast_langdetect for language detection Thanks for any guidance! 🙏
r
The
tts_node
method receives an
AsyncIterable[str]
of text chunks from the LLM. tts_node.py:38-52 You can process this stream to detect language before forwarding to TTS:
Copy code
async def tts_node(self, text: AsyncIterable[str], model_settings: ModelSettings):  
    async def process_text():  
        accumulated_text = ""  
        async for chunk in text:  
            accumulated_text += chunk  
              
            # Detect language once you have enough text  
            if len(accumulated_text.strip()) >= 20:  # threshold  
                detected_lang = detect(accumulated_text)[0]['lang']  
                if detected_lang != self.current_language:  
                    self.tts.update_options(  
                        language=detected_lang,  
                        voice_id=LANGUAGE_VOICE_MAPPING[detected_lang]  
                    )  
                    self.current_language = detected_lang  
              
            yield chunk  
      
    return Agent.default.tts_node(self, process_text(), model_settings)