Thanks — good question. You can replace an STT step with an audio‑LLM and have the model accept audio directly and execute prompts in real time. LiveKit Agents supports realtime multimodal models that take audio input (listen), reason, and produce responses (audio and/or text) without an intermediate transcription step. What this enables • End-to-end realtime voice agents: user audio is streamed from the browser (WebRTC) through the agent to a realtime multimodal model, which processes audio directly and returns responses immediately. • Simpler pipeline: no separate STT + LLM + TTS chain — the model handles perception, reasoning, and response generation. • Better latency and interruption handling: realtime models can stream responses and handle user interruptions/push-to-talk patterns more naturally. Which models / integrations are supported • Realtime multimodal models such as OpenAI Realtime, Azure OpenAI Realtime, Gemini Live, and Ultravox Realtime are usable with LiveKit Agents and accept direct audio input. • LiveKit’s Realtime model integration (for example

openai.realtime.RealtimeModel()

in an agent session) lets you stream audio to the model and receive audio/text output without requiring an STT plugin. Follow-ups I can help with • If you want, I can outline a minimal architecture for streaming audio from WebRTC to a realtime model and returning responses. • I can show example steps or snippets for setting up an agent session that uses a realtime multimodal model. • Tell me which model/provider you plan to use (OpenAI, Azure, Gemini, Ultravox, etc.) and whether you need audio-only responses or both audio + text — that will let me give more targeted instructions. Sources: Realtime models overview | LiveKit Documentation | OpenAI and LiveKit partner to turn Advanced Voice into an API | examples/voice_agents | livekit/livekit_composite | DeepWiki | Improving voice AI's turn detection with transformers