I m trying to set up my Gemini voice agent to use vertex AI LiveKit Community #ask-ai

I'm trying to set up my Gemini voice agent to use ...

breezy-art-50071

12/22/2025, 1:06 PM

I'm trying to set up my Gemini voice agent to use vertex AI. I have this agent code:

Copy code

from dotenv import load_dotenv

from livekit import agents, rtc
from livekit.agents import AgentServer, AgentSession, Agent, room_io
from livekit.plugins import (
    google,
    noise_cancellation,
    silero,
)

load_dotenv("staging.env")


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful voice AI assistant powered by Gemini.")


server = AgentServer()


@server.rtc_session()
async def my_agent(ctx: agents.JobContext):
    session = AgentSession(
        llm=google.realtime.RealtimeModel(
            model="gemini-2.0-flash-exp",
            voice="Puck",
            vertexai=True
        ),
        vad=silero.VAD.load(),
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
        room_options=room_io.RoomOptions(
            audio_input=room_io.AudioInputOptions(
                noise_cancellation=lambda params: noise_cancellation.BVCTelephony() if params.participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP else noise_cancellation.BVC(),
            ),
        ),
    )

    await session.generate_reply(
        instructions="Greet the user and offer your assistance."
    )


if __name__ == "__main__":
    agents.cli.run_app(server)

And my .env file has the following:

Copy code

GOOGLE_API_KEY="<some-key>"
ENVIRONMENT=staging
LIVEKIT_PROJECT=<project-name>
GOOGLE_APPLICATION_CREDENTIALS=<path-to-json-file>
GOOGLE_CLOUD_PROJECT=<project-id>
GOOGLE_CLOUD_LOCATION=europe-north2
# remove
LIVEKIT_API_KEY=<some-key>
LIVEKIT_API_SECRET=<some-key>
LIVEKIT_URL=<some-url>

When I try to start it in dev mode, I get the following error:

Copy code

13:27:29.468 ERROR  livekit.plugins… Gemini Realtime API error: server rejected WebSocket connection: HTTP 404                                                          
                                         Traceback (most recent call last):                                                                                                 
                                           File                                                                                                                             
                                         "/Users/timothyakoji/.pyenv/versions/popscale-agent/lib/python3.12/site-packages/livekit/plugins/google/realtime/realtime_api.py"  
                                         , line 667, in _main_task                                                                                                          
                                             async with self._client.aio.live.connect(                                                                                      
                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                      
                                           File "/Users/timothyakoji/.pyenv/versions/3.12.12/lib/python3.12/contextlib.py", line 210, in __aenter__                         
                                             return await anext(self.gen)                                                                                                   
                                                    ^^^^^^^^^^^^^^^^^^^^^                                                                                                   
                                           File "/Users/timothyakoji/.pyenv/versions/popscale-agent/lib/python3.12/site-packages/google/genai/live.py", line 1079, in       
                                         connect                                                                                                                            
                                             async with ws_connect(                                                                                                         
                                                        ^^^^^^^^^^^                                                                                                         
                                           File "/Users/timothyakoji/.pyenv/versions/popscale-agent/lib/python3.12/site-packages/websockets/asyncio/client.py", line 587,   
                                         in __aenter__                                                                                                                      
                                             return await self                                                                                                              
                                                    ^^^^^^^^^^                                                                                                              
                                           File "/Users/timothyakoji/.pyenv/versions/popscale-agent/lib/python3.12/site-packages/websockets/asyncio/client.py", line 543,   
                                         in __await_impl__                                                                                                                  
                                             await self.connection.handshake(                                                                                               
                                           File "/Users/timothyakoji/.pyenv/versions/popscale-agent/lib/python3.12/site-packages/websockets/asyncio/client.py", line 114,   
                                         in handshake                                                                                                                       
                                             raise self.protocol.handshake_exc                                                                                              
                                           File "/Users/timothyakoji/.pyenv/versions/popscale-agent/lib/python3.12/site-packages/websockets/client.py", line 325, in parse  
                                             self.process_response(response)                                                                                                
                                           File "/Users/timothyakoji/.pyenv/versions/popscale-agent/lib/python3.12/site-packages/websockets/client.py", line 142, in        
                                         process_response                                                                                                                   
                                             raise InvalidStatus(response)                                                                                                  
                                         websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 404

How can I get the Vertex AI integration to work on my voice agent?

tall-airline-16756

12/22/2025, 1:07 PM

Thanks for reaching out, Timothy. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

tall-airline-16756

12/22/2025, 1:07 PM

Thanks — you're hitting the WebSocket 404 because

RealtimeModel

(Gemini Live) does not work through Vertex AI. I looked at your code and env — you set

vertexai=True

and provided

GOOGLE_APPLICATION_CREDENTIALS

, which is for Vertex AI, but

google.realtime.RealtimeModel

talks to the Gemini Live API (generativelanguage endpoint) and requires a Gemini API key instead. What to change (quick steps): 1. Remove

vertexai=True

from `RealtimeModel`:

Copy code

llm=google.realtime.RealtimeModel(
    model="gemini-2.0-flash-exp",
    voice="Puck",
    # remove vertexai=True
),

1. Use a Google API key (Gemini / Generative Language API) in your env: • Keep

GOOGLE_API_KEY=<your_api_key>

• Remove or ignore

GOOGLE_APPLICATION_CREDENTIALS

GOOGLE_CLOUD_PROJECT

, and

GOOGLE_CLOUD_LOCATION

for this realtime path. 1. Ensure the Generative Language (Gemini) API is enabled for the project that issued the API key and the key allows access to generativelanguage.googleapis.com. Why this fixes the 404: • The Gemini Live API expects connections via the Gemini endpoint using an API key. Vertex AI does not expose the Gemini Live (real-time audio) endpoint, so setting

vertexai=True

causes the client to try the wrong routing and the server returns 404. If you want Vertex AI instead (service account auth,

GOOGLE_APPLICATION_CREDENTIALS

), you must use non-realtime LLM flows (e.g.,

google.LLM(..., vertexai=True)

) with a separate STT→LLM→TTS pipeline — not

RealtimeModel

. Would you like me to: • Validate your .env values (you can paste them with secrets redacted), or • Show a full minimal working example for Gemini realtime voice with the correct env? Sources: Google Gemini LLM plugin guide | LiveKit docs | Gemini Live API plugin | LiveKit docs

2 Views

Open in Slack

Previous Next