Hi I am using google realtime model as LLM Do we need to add seprate TTS STT? or we can have NOT_GIVEN? If we add STT and TTS will it use that?

Thanks for reaching out, Madhav. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

yes I am using livekit builtin feature I am asking if livekit support TTS and STT with Google realtime model

I'm happy to share that LiveKit natively supports TTS (Text-to-Speech) and STT (Speech-to-Text) integration with Google's Realtime Model (Gemini Live API) through its built-in features. This means you can enjoy real-time, end-to-end speech-to-speech interactions without needing separate TTS or STT components. If you’d like a bit more control over the audio output, such as customizing voices, you can also set up the system to use a separate TTS plugin. In this setup, the Realtime Model can return text only, and the TTS plugin will take care of the audio output for you. Please feel free to reach out if you'd like detailed setup instructions or any further help—I’m here to support you! Sources: Realtime model integrations | LiveKit Docs | livekit-plugins/livekit-plugins-google/livekit/plugins/google/beta/realtime... | Google AI and LiveKit | LiveKit Docs | Google integration guide | LiveKit Docs | Google integration guide | LiveKit Docs

2025-09-18 151957,098 - ERROR livekit.plugins.google - Gemini Realtime API error: received 1007 (invalid frame payload data) Cannot extract voices from a non-audio request.; then sent 1007 (invalid frame payload data) Cannot extract voices from a non-audio request. Traceback (most recent call last): File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/livekit/plugins/google/beta/realtime/realtime_api.py”, line 591, in _main_task async with self._client.aio.live.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/opt/anaconda3/lib/python3.12/contextlib.py”, line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/google/genai/live.py”, line 1057, in connect logger.info(await ws.recv(decode=False)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/websockets/asyncio/connection.py”, line 322, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: received 1007 (invalid frame payload data) Cannot extract voices from a non-audio request.; then sent 1007 (invalid frame payload data) Cannot extract voices from a non-audio request. {“pid”: 69282, “job_id”: “AJ_WHzd7bT7GP9Q”} 2025-09-18 151957,112 - ERROR livekit.agents - AgentSession is closing due to unrecoverable error Traceback (most recent call last): File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/livekit/plugins/google/beta/realtime/realtime_api.py”, line 591, in _main_task async with self._client.aio.live.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/opt/anaconda3/lib/python3.12/contextlib.py”, line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/google/genai/live.py”, line 1057, in connect logger.info(await ws.recv(decode=False)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/websockets/asyncio/connection.py”, line 322, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: received 1007 (invalid frame payload data) Cannot extract voices from a non-audio request.; then sent 1007 (invalid frame payload data) Cannot extract voices from a non-audio request. {“pid”: 69282, “job_id”: “AJ_WHzd7bT7GP9Q”} 2025-09-18 151957,114 - ERROR livekit.plugins.google - Error in _main_task Traceback (most recent call last): File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/livekit/plugins/google/beta/realtime/realtime_api.py”, line 591, in _main_task async with self._client.aio.live.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/opt/anaconda3/lib/python3.12/contextlib.py”, line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/google/genai/live.py”, line 1057, in connect logger.info(await ws.recv(decode=False)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/websockets/asyncio/connection.py”, line 322, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: received 1007 (invalid frame payload data) Cannot extract voices from a non-audio request.; then sent 1007 (invalid frame payload data) Cannot extract voices from a non-audio request. The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/livekit/agents/utils/log.py”, line 16, in async_fn_logs return await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/livekit/plugins/google/beta/realtime/realtime_api.py”, line 641, in _main_task raise APIConnectionError(message=“Failed to connect to Gemini Live”) from e livekit.agents._exceptions.APIConnectionError: Failed to connect to Gemini Live (body=None, retryable=True) {“pid”: 69282, “job_id”: “AJ_WHzd7bT7GP9Q”} 2025-09-18 151959,482 - ERROR livekit.agents - Error in _realtime_reply_task Traceback (most recent call last): File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/livekit/agents/utils/log.py”, line 16, in async_fn_logs return await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/madhavsharma/uplevel-livekit-worker/venv/lib/python3.12/site-packages/livekit/agents/voice/agent_activity.py”, line 1859, in _realtime_reply_task generation_ev = await self._rt_session.generate_reply( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I'm sorry to hear you're encountering the error "received 1007 (invalid frame payload data) Cannot extract voices from a non-audio request." This usually happens when the Gemini Realtime API gets non-audio data in a request that's expecting audio input. Let's work through some steps together to help you resolve this: 1. Ensure proper audio format: The API expects audio input at a 16kHz sample rate and 1 channel. It’s a good idea to double-check that your input audio matches this format. 2. Verify audio frame integrity: Make sure the

rtc.AudioFrame

objects you're passing to

push_audio()

contain valid audio data and aren’t empty or corrupted. 3. Check for proper audio transmission: Since the API processes audio in 50ms chunks, ensure your audio source consistently provides frames without any interruptions. 4. Validate the audio pipeline: If you’re using a custom audio source, please confirm it’s properly configured and capturing frames correctly. 5. Handle audio resampling: If your input audio changes sample rates dynamically, verify that the

AudioResampler

is managing this correctly. To dig a little deeper, you might also try: • Logging audio frames before pushing them to

push_audio()

to ensure they contain valid PCM data. • Checking the audio resampler configuration if your input format differs from the required 16kHz/1-channel format. • Increasing the

queue_size_ms

parameter in your

AudioSource

to help prevent buffer overflows. If you’d like, I’m here to guide you through these steps or answer any other questions you have. Just let me know! Sources: livekit-plugins/livekit-plugins-google/livekit/plugins/google/beta/realtime... | plugins/google/src/beta/realtime/realtime_api.ts | Resolving "InvalidState - failed to capture frame" errors in AudioSource