Thanks — this is clear and helpful context. I’ll respond about how to stop the STT/TTS pipeline from activating in Text Mode and prevent the synthesize warning. Acknowledging your question • You want the agent to keep STT and TTS off in Text Mode, and only enable STT/TTS when switching to Voice Mode. Right now the pipelines are starting even in Text Mode and producing: WARNINGlivekit.agentsfailed to synthesize speech: Connection error., retrying in 0.1s What to do (best, supported approach) • Use the agent session audio I/O toggles to explicitly disable both input (STT) and output (TTS) when in Text Mode: ◦ To disable STT: session.input.set_audio_enabled(False) (or session.input.setAudioEnabled(false) in TypeScript) ◦ To disable TTS: session.output.set_audio_enabled(False) (or session.output.setAudioEnabled(false) in TypeScript) Why this fixes the problem • Disabling

session.input

prevents the agent from subscribing/processing audio input, so STT will not run. • Disabling

session.output

prevents TTS synthesis and publishing of audio, so the agent will not attempt to call the speech synthesis service (and therefore avoids the connection error and retry loop). How to wire this into your state machine 1. Text Mode (default) ◦ Call: ▪︎ session.input.set_audio_enabled(False) ▪︎ session.output.set_audio_enabled(False) ◦ Result: STT and TTS both off. 1. Transition Text Mode → Voice Mode (frontend enables microphone) ◦ On agent: when you receive the track_subscribed / track_published event, call: ▪︎ session.input.set_audio_enabled(True) ▪︎ session.output.set_audio_enabled(True) ◦ Result: STT and TTS both on. 1. Voice Mode → Text Mode (frontend disables microphone) ◦ On agent: when you receive the track_unsubscribed / track_unpublished event, call: ▪︎ session.input.set_audio_enabled(False) ▪︎ session.output.set_audio_enabled(False) ◦ Result: STT and TTS both off. 1. Mic Mute (while in Voice Mode) ◦ On agent: when you receive track_muted, call: ▪︎ session.input.set_audio_enabled(False) (STT off) ▪︎ Keep session.output.set_audio_enabled(True) if you want TTS responses to continue (agent speaks even with mic muted). ◦ On track_unmuted: re-enable input via session.input.set_audio_enabled(True). Practical notes • Just relying on “no audio being published from the frontend” is not enough if

session.input

remains enabled. The agent can still attempt to run input processing and the output pipeline can still attempt TTS. • Turning

session.output

off is the key to stop TTS attempts (and the synthesize connection errors). • Use your room event handlers (track_published / track_unpublished / track_muted / track_unmuted) to toggle these session settings automatically. If you’d like, I can: • Provide a short code snippet in your preferred language (Python or TypeScript) that shows the event listeners and the exact calls to set_audio_enabled. • Help map any existing frontend events you already emit into the correct agent-side toggles. Which language is your agent runtime in (Python or TypeScript)? Sources: Text and transcriptions | LiveKit Documentation | Agent session | LiveKit Documentation