Thanks — I looked over your code and the short answer is: your intent is fine, but the current implementation is not correct. I’ll highlight the main problems and give a concise, correct approach you can follow. What I’m responding to • Your question about whether switching agent behavior by instantiating a new agent and calling

session.update_agent()

when a participant connects is correct. Main problems in your code • You have two different

async def entrypoint(ctx: JobContext):

definitions. Python will overwrite the first one — keep a single entrypoint. • You create a new agent instance (e.g.,

Transcriber()

) and call

await session.update_agent(agent=transcriber_agent)

without ensuring that agent is initialized with the same session context/plugins (STT/LLM/TTS) the session uses. • Your

Transcriber

is created without passing/using the session-level STT/LLM/TTS configuration. Agents should be initialized consistently with the session’s configured plugins or the agent should inherit them from the session. • Switching agents mid-session must coordinate lifecycle/state. Directly instantiating an agent and swapping it in may cause missing state, missing plugin config, or unexpected behavior. • You have two different STT plugin usages (assemblyai and azure) mixed; be intentional which STT provider you want to use for speaker ID/diarization. What you can do instead (recommended) 1. Use a single

entrypoint(ctx: JobContext)

and do all session/agent setup there. 2. Configure your session with the STT/LLM/TTS/VAD you want once, and create agents that are constructed to use those plugins (either by passing them in or relying on the session). 3. When you need to change agent behavior at runtime, call

await session.update_agent(new_agent)

only if

new_agent

is correctly initialized and intended to run in the current session context. 4. Alternatively, use an LLM/tool-based handoff (return a new agent from a tool or use an explicit handoff mechanism) so the transition is handled inside the agent lifecycle logic rather than directly from a connection handler. Minimal corrected structure (conceptual) • Single entrypoint • Build STT/LLM/TTS (with configs) once • Create an AgentSession with those plugins • Start session with initial agent (e.g., SummaryAgent) • In participant-connected handler, only switch if: ◦ You create the new agent with correct plugin config, or ◦ The new agent is designed to run without additional plugins (e.g., a pure transcriber that only needs session STT) Example sketch (conceptual): 1. Build STT/LLM/TTS (e.g.,

stt = assemblyai.STT()

or

azure.STT(...)

) 2. session = AgentSession(stt=stt, llm=..., tts=..., vad=...) 3. await session.start(agent=SummaryAgent(), room=ctx.room, ...) 4. @ctx.room.on("participant_connected") if participant.identity == "transcribe_user": transcriber = Transcriber() # ensure it's compatible with session plugins await session.update_agent(transcriber) Speaker ID / diarization notes • Speaker diarization depends on the STT provider: ensure the STT plugin you use exposes per-speaker labels in transcription events. • If you rely on Speechmatics (or another provider), confirm the plugin emits speaker IDs on transcription events, then consume those events to associate text with speakers. • If your plugin does not emit speaker data, you’ll need an STT that supports diarization or a separate diarization step. If you’d like, I can: • Provide a corrected, runnable example using a single STT provider (AssemblyAI, Azure, or Speechmatics) and a single entrypoint. • Show how to consume speaker-labeled transcription events from your chosen STT plugin. Which STT provider are you planning to use for speaker ID/diarization (Speechmatics, AssemblyAI, Azure, or another)? Sources: Agents and handoffs | LiveKit Documentation | TTS Translator with Gladia STT | LiveKit Documentation | ElevenLabs Change Language | LiveKit Documentation | TTS Comparison | LiveKit Documentation