When using generate_reply() with google gemini liv...
# agents
a
When using generate_reply() with google gemini live model I get error:
Copy code
2025-11-13 14:45:20,074 - ERROR livekit.agents - Error in _realtime_reply_task
Traceback (most recent call last):
  File "/Users/malteholledig/Freja/freja-agent-python/venv/lib/python3.13/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malteholledig/Freja/freja-agent-python/venv/lib/python3.13/site-packages/livekit/agents/voice/agent_activity.py", line 1942, in _realtime_reply_task
    generation_ev = await self._rt_session.generate_reply(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        instructions=instructions or NOT_GIVEN
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
livekit.agents.llm.realtime.RealtimeError: generate_reply timed out waiting for generation_created event.
I am using it the correct place and works when using openAI realtime. Is this intended not to work with gemini or is it a bug? FYI i am using latest package versions. @refined-appointment-81829 any ideas?
l
I tried this out with our minimal example at https://github.com/livekit-examples/python-agents-examples/blob/main/realtime/gemini_realtime_api.py and it shows the same behaviour. The code sample shown at https://docs.livekit.io/agents/models/realtime/plugins/gemini/ does not give that error. The key difference seems to be
proactivity=True
, if this is set to false,
generate_reply()
works as expected. Do you have this setting set?
a
So when I set
proactivity=False
, I get:
Copy code
ERROR agent - Session is closing due to unrecoverable error type='realtime_model_error' timestamp=1763116622.887485 label='livekit.plugins.google.realtime.realtime_api.RealtimeModel' error=ConnectionClosedError(Close(code=1007, reason='Invalid JSON payload received. Unknown name "proactivity" at \'setup\': Cannot find field.'), Close(code=1007, reason='Invalid JSON payload received. Unknown name "proactivity" at \'setup\': Ca
Seems like a bug cause I dont get it when I set it to
proactivity=True
but here I get:
Copy code
2025-11-14 11:38:31,090 - ERROR livekit.agents - Error in _realtime_reply_task
Traceback (most recent call last):
  File "/Users/malteholledig/Freja/freja-agent-python/venv/lib/python3.13/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malteholledig/Freja/freja-agent-python/venv/lib/python3.13/site-packages/livekit/agents/voice/agent_activity.py", line 1942, in _realtime_reply_task
    generation_ev = await self._rt_session.generate_reply(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        instructions=instructions or NOT_GIVEN
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
livekit.agents.llm.realtime.RealtimeError: generate_reply timed out waiting for generation_created event. {"pid": 18225, "job_id": "AJ_efhBfCm46bku"}
When I completely remove it I still get above error but with these warnings first:
Copy code
2025-11-14 11:38:26,652 - WARNING livekit.plugins.google - received server content but no active generation. {"pid": 18225, "job_id": "AJ_efhBfCm46bku"}
2025-11-14 11:38:26,652 - WARNING livekit.plugins.google - received server content but no active generation. {"pid": 18225, "job_id": "AJ_efhBfCm46bku"}
2025-11-14 11:38:26,653 - WARNING livekit.plugins.google - no active generation to report metrics for {"pid": 18225, "job_id": "AJ_efhBfCm46bku"}
l
Sorry, I should have been clearer, I was deleting the proactivity setting rather than setting it to False. What I did was to take https://github.com/livekit-examples/python-agents-examples/blob/main/realtime/gemini_realtime_api.py, then modify it as follows:
Copy code
session = AgentSession(
        llm=google.realtime.RealtimeModel(
            model="gemini-2.5-flash-native-audio-preview-09-2025",
            voice="Puck",
            temperature=0.8,
            instructions="You are a helpful assistant.",
            enable_affective_dialog=True
        ),
        vad=silero.VAD.load()
    )
I am also using version 1.2.18 of livekit-agents and the livekit-plugins-google. Can you match my setup and see if you still see the issue? It looks like there's some issue with the example, but I want to get something working for you first.
a
Ah thanks. I was able to make that example work. And reimplementing it to my solution it seems like it stopped happening when adding this one to the session config:
enable_affective_dialog=True
However that introduces another issue where it executes my "hang_up" tool when initialising - which basically means it "ends the call" just after connecting. Not at all sure how that can be connected to the affective dialog but only happens when I have it enabled.. 🤔
So removing the
enable_affective_dialog=True
from your example reintroduces the issue. Dunno it you are able to reproduce?
👀 1
I was using this one in my session config
Copy code
input_audio_transcription=InputAudioTranscription(
                model="gpt-4o-transcribe",
                language=language,
                prompt=f"Expect {get_language_name(language)} words and sentences. Context is a phone call.",)
Disabling that seemed to fix the instant tool call with affective dialog issue.
l
Glad you are unblocked, I see Google has warnings on both those flags that they can produce unreliable results, I'll remove them from our demo.