I wanted to clarify how to best manage the context for the gemini live api model. i have multiple different contexts in a db. the user of my applocation will choose a specific one to talk to the voice agent about. What is the recommended approach for pulling something from a db and using it in the agent context / system instruction

Thanks for reaching out, adit-kulkarni. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

i guess what im trying to ask is: say my user defines a topic of discussion "i want to talk about stars" - how do I pass this to the deployed agent?

Thanks — you want to pass a user-selected topic like “I want to talk about stars” into a deployed Gemini Live voice agent so the agent uses it as the conversation context. Here are two practical, safe ways to do that and when to use each. Short answer • Put the topic into the model’s

instructions

if it should define the agent’s role/behavior for the whole session. • Or inject the topic as an initial chat message /

ChatContext

if it’s transient or part of the ongoing conversation. Option A — Make the topic the session’s system instruction (recommended if the topic defines the session) 1. Read the user’s selection on your server:

topic = "I want to talk about stars"

. 2. Build a concise instruction that frames the agent role and how to use the topic: ◦ Example instruction: "You are an educational voice assistant. The user wants to discuss: stars. Keep answers friendly and high-level unless asked for technical detail." 1. Create the AgentSession / RealtimeModel with that string as

instructions

before connecting:

context = f"You are an educational voice assistant. The user wants to discuss: {topic}."
session = AgentSession(
  llm=google.realtime.RealtimeModel(
    model="gemini-2.5-flash",
    instructions=context,
    voice="Puck"
  ),
  vad=...
)
# then connect the session

When to use: topic should persist for the whole call and shape the agent’s persona/limits. Option B — Send the topic as the initial chat message / ChatContext (recommended if topic is conversational or may change) 1. Fetch the topic, then add it as the first user message or chat-context entry:

# pseudo-structure
initial_messages = [
  {"role":"system","content":"You are a helpful voice assistant."},
  {"role":"user","content":"I want to talk about stars"}
]
# pass initial_messages into ChatContext or as initial conversation state

1. Start the session with the system prompt minimal and include the initial messages so the model treats it as current conversation context. When to use: topic is one of many conversation inputs, may change mid-call, or you prefer not to bake content into permanent instructions. Practical tips • Keep

instructions

concise and focused on role/constraints; avoid dumping very long docs there. • If topic implies long source material, summarize or attach snippets as chat messages (RAG) rather than a huge system prompt. • Ensure DB lookup completes before creating/connecting the session to avoid the agent starting without context. • Sanitize user input (remove PII or unsafe content) before injecting it into prompts. If you want, tell me: • whether the topic should persist for the entire session or can change mid-call, and • the typical length of topic/context text — I’ll suggest exact prompt templates or a RAG pattern for long contexts.

hmm im not sure you understand. let's say i want to have dynamic assistants whose instructions are set by the user, how would i pass the information ffrom my frontend to the agent i have deployed?

Thanks — I see you want to let end users define assistants (their instructions) in the frontend and have those instructions applied to an already-deployed Gemini Live voice agent. A few quick clarifying questions so I give the exact pattern you need: • Do you need to set the instructions only at session start, or update them mid-call (live)? • How does your frontend talk to your backend today? (HTTP REST call, WebSocket, direct to LiveKit, etc.) • Is the agent deployed as a long-running service that accepts session creation requests, or do you create an AgentSession per user connection on your server? If you tell me those three details I’ll provide a concise, step-by-step implementation pattern (frontend payload → backend handling → where to place the text:

instructions

, initial chat message/ChatContext, or room/participant metadata) and example snippets tailored to your stack.

the instructions are set at the start. my frontend will communicate with livekit cloud via web rtc which will route my request to the agent i have deployed on livekit cloud. i want to know how i can pass the insturction data from my frontend to the agent

Thanks — you're asking how to pass user-defined assistant instructions from your frontend (WebRTC → LiveKit Cloud) into a deployed agent at session start. Here's the concise, correct pattern to do that. Short flow (frontend → backend → LiveKit dispatch → agent): 1. Frontend: collect the user’s instruction text (e.g., "I want to talk about stars") and POST it to your backend. 2. Backend: call LiveKit's agent dispatch API and include the instruction in the dispatch request's

metadata

(JSON string). 3. LiveKit / Agent: in the agent entrypoint, read

ctx.job.metadata

, parse the JSON, and inject the value into the model as

instructions

(or otherwise apply it when creating the AgentSession) before calling

ctx.connect()

. Why this works • WebRTC carries media; metadata must be sent via your backend using the LiveKit dispatch API so the agent receives it at startup. • The agent reads

ctx.job.metadata

at entry and can set the Gemini Live model

instructions

to the user-provided text for the whole session. Minimal examples • Backend dispatch (Node-ish pseudocode):

// after receiving topic from frontend
const metadata = JSON.stringify({ custom_instructions: userText });
await dispatchClient.createDispatch("room-name", "agent-name", { metadata });

• Agent entrypoint (pseudo-Python):

async def entrypoint(ctx: JobContext):
    metadata = json.loads(ctx.job.metadata or "{}")
    instr = metadata.get("custom_instructions", "You are a helpful assistant.")
    session = AgentSession(
      llm=google.realtime.RealtimeModel(model="gemini-2.5-flash", instructions=instr),
      ...
    )
    await session.start(room=ctx.room, agent=...)
    await ctx.connect()

Practical tips • Ensure backend completes the dispatch (with metadata) before the client joins the room so the agent starts with instructions in place. • Keep instructions concise; summarize or chunk large content. • Sanitize/redact user input as needed before embedding into

instructions

. If you want, tell me which backend language or SDK you use and I’ll give a concrete code snippet you can drop in. Sources: Agent dispatch | LiveKit docs | Job lifecycle | LiveKit docs | Context Variables | LiveKit docs

is it better to do it your way via dispatching an agent or can i pass data via room.localParticipant.set(metadata)

Good question — both can carry data, but for setting session-level agent instructions at startup, dispatching the agent with metadata is the better, more reliable approach. Quick comparison and why: Why dispatch metadata is recommended • Delivered to the agent at creation: metadata in the dispatch request is available to the agent in

ctx.job.metadata

during its entrypoint, so the agent can set

instructions

before calling

ctx.connect()

. • Guaranteed timing: the agent receives the instructions before it joins the room, avoiding race conditions where the agent starts without the right context. • Intended pattern: dispatch metadata is the explicit mechanism for passing session-level configuration to agents on LiveKit Cloud. • Cleaner security/control: your backend controls and validates the instructions before dispatching. Why

room.localParticipant.set(metadata)

is less suitable for this use case • Participant metadata is propagated over the room and is meant for participant-level info, not guaranteed startup config for agents. • Timing/race risk: the agent may already be initialized before that metadata arrives or before the agent reads participant attributes. • Less direct for agents: the agent entrypoint reads job metadata easily; reading participant metadata would require extra logic to watch for updates and may complicate startup flow. When participant metadata can be useful • Passing per-user presence info, display names, or transient participant attributes that both clients and the agent can read during an ongoing session. • Small live updates mid-call (though agents may still need explicit handling to consume them). Recommendation • Use dispatch metadata for session-start instructions (set

instructions

from

ctx.job.metadata

). • Use participant metadata only for user/participant-level state or non-critical live updates. If you want, tell me whether you need to support updating instructions before the user joins the room (but after dispatch) or to change them mid-session — I can show exact code patterns for safe ordering or for listening to participant metadata changes.