https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# ask-ai
s
This message was deleted.
t
Thanks for reaching out, Lasha. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — you're asking how to give your agent a dynamic first message using the Google Realtime model. You can do this by passing a runtime, personalized instruction when you create/start the Realtime session and then generating the first reply with that instruction. Quick steps: 1. Build a dynamic instruction string (template + runtime variables like user name, location, etc.). 2. Create the RealtimeModel with that instruction (or pass it when starting the session). 3. Start the AgentSession and immediately call 
generate_reply()
with the dynamic instruction to send the first message. Minimal Python example:
Copy code
@server.rtc_session()
async def entrypoint(ctx: JobContext):
    user = get_user_context(ctx)  # e.g., {'name': 'Alex', 'city': 'Berlin'}
    dynamic_instructions = (
        f"You are a helpful assistant for {user['name']} from {user['city']}. "
        "Greet them by name and offer personalized help."
    )

    session = AgentSession(
        llm=google.beta.realtime.RealtimeModel(
            model="gemini-2.5-flash-native-audio-preview-09-2025",
            instructions=dynamic_instructions
        ),
        vad=ctx.proc.userdata["vad"],
    )

    await session.start(agent=Assistant(), room=ctx.room)
    await ctx.connect()

    # Send the personalized first message
    await session.generate_reply(instructions=dynamic_instructions)
If you want alternatives: • Pass a lightweight system prompt in 
RealtimeModel
and supply richer runtime instructions in 
generate_reply()
. • Use a template engine to render complex prompts from user metadata before starting the session. Which approach do you prefer — setting the prompt at session creation or sending it only via 
generate_reply()
? If you share a short example of your current code and the variables you have at runtime, I’ll adapt the snippet to your setup. Sources: Gemini Realtime Agent with Live Vision | LiveKit docs | examples/voice_agents | livekit/livekit_composite | DeepWiki
k
import os from dotenv import load_dotenv import asyncio import logging from livekit import agents from livekit.agents import Agent, AgentSession, JobContext from livekit.plugins import google, silero load_dotenv(".env.local") class Assistant(Agent): def __init__(self) -> None: super().__init__(instructions="გამარჯობა! მე ვარ პიცის მიტანის AI ასისტენტი. რით შემიძლია დაგეხმაროთ?") async def on_enter(self): # Static first message when agent joins await self.session.generate_reply( instructions="გამარჯობა! მე ვარ პიცის მიტანის AI ასისტენტი. რით შემიძლია დაგეხმაროთ?" ) async def entrypoint(_ctx_: JobContext): await ctx.connect() # Add voice plugins session = AgentSession( llm=google.realtime.RealtimeModel( model="gemini-2.5-flash-native-audio-preview-09-2025", voice="Kore" ), vad=silero.VAD.load() ) await session.start(agent=Assistant(), room=ctx.room) if name == "__main__": agents.cli.run_app( agents.WorkerOptions( _entrypoint_fnc_=entrypoint, _agent_name_="test_agent", ) ) This is my testing code, if that will help.
t
Thanks — this is helpful. I’ll show the minimal changes to make the first message dynamic using your provided code. Key change: render a dynamic instruction string from runtime context (e.g., user name) and pass it to 
AgentSession
or to 
generate_reply()
right after session start. Modified example (only changed/added lines shown and indent fixed):
Copy code
# inside your Assistant class
class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="გამარჯობა! მე ვარ პიცის მიტანის AI ასისტენტი. რით შემიძლია დაგეხმაროთ?")

    async def on_enter(self):
        # removed static send here — we'll send a dynamic first message from entrypoint
        pass

# in entrypoint
async def entrypoint(ctx: JobContext):
    await ctx.connect()

    # Example dynamic context you have at runtime
    user = {"name": "ნინო", "city": "Tbilisi"}  # replace with real runtime values

    dynamic_instructions = (
        f"გამარჯობა {user['name']} from {user['city']}! "
        "მე ვარ პიცის მიტანის AI ასისტენტი. რით შემიძლია დაგეხმაროთ?"
    )

    session = AgentSession(
        llm=google.realtime.RealtimeModel(
            model="gemini-2.5-flash-native-audio-preview-09-2025",
            voice="Kore",
            instructions=dynamic_instructions,  # set initial system prompt here (optional)
        ),
        vad=silero.VAD.load(),
    )

    await session.start(agent=Assistant(), room=ctx.room)
    await ctx.connect()

    # Send the personalized first message explicitly
    await session.generate_reply(instructions=dynamic_instructions)
Notes: • You can set the dynamic prompt either in 
RealtimeModel(..., instructions=...)
or pass it to 
generate_reply(instructions=...)
. Passing to 
generate_reply()
is useful if you want different first messages per user without recreating the model. • Replace the 
user
dictionary with whatever runtime data you have (session metadata, query params, etc.). • If you want the assistant to speak the message, keep 
voice="Kore"
and ensure the client handles audio output. Would you like a version that reads user context from environment variables, HTTP request metadata, or LiveKit room metadata? Which runtime variables do you have available?
k
I think your code Minimal Python example:
Copy code
@server.rtc_session()
async def entrypoint(ctx: JobContext):
    user = get_user_context(ctx)  # e.g., {'name': 'Alex', 'city': 'Berlin'}
    dynamic_instructions = (
        f"You are a helpful assistant for {user['name']} from {user['city']}. "
        "Greet them by name and offer personalized help."
    )

    session = AgentSession(
        llm=google.beta.realtime.RealtimeModel(
            model="gemini-2.5-flash-native-audio-preview-09-2025",
            instructions=dynamic_instructions
        ),
        vad=ctx.proc.userdata["vad"],
    )

    await session.start(agent=Assistant(), room=ctx.room)
    await ctx.connect()

    # Send the personalized first message
    await session.generate_reply(instructions=dynamic_instructions)
Is better, can you adjust it to my needs? Can you give me simple agent.py?
t
Below is a simple, ready-to-run agent.py adapted from your code that sends a dynamic first message using runtime user context (reads from LiveKit room metadata if available, falls back to env vars). It uses the Google Realtime model and Silero VAD like your example. Save as agent.py and update your .env.local with any needed keys.
Copy code
import os
from dotenv import load_dotenv
import asyncio
import logging
from livekit import agents
from livekit.agents import Agent, AgentSession, JobContext
from livekit.plugins import google, silero

load_dotenv(".env.local")
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)

def get_user_context_from_room(ctx: JobContext):
    # Try to read simple room metadata; adapt to your actual metadata keys
    md = getattr(ctx.room, "metadata", None) or {}
    # Example keys: name, city
    return {
        "name": md.get("user_name") or os.getenv("DEFAULT_USER_NAME", "Guest"),
        "city": md.get("user_city") or os.getenv("DEFAULT_USER_CITY", "your city"),
    }

class Assistant(Agent):
    def __init__(self) -> None:
        # Keep a default instruction; dynamic greeting will be sent at runtime.
        super().__init__(instructions="გამარჯობა! მე ვარ პიცის მიტანის AI ასისტენტი. რით შემიძლია დაგეხმაროთ?")

    async def on_enter(self):
        # No static message here; the entrypoint will send the dynamic first message.
        return

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    user = get_user_context_from_room(ctx)
    dynamic_instructions = (
        f"გამარჯობა {user['name']} from {user['city']}! "
        "მე ვარ პიცის მიტანის AI ასისტენტი. რით შემიძლია დაგეხმაროთ?"
    )

    session = AgentSession(
        llm=google.realtime.RealtimeModel(
            model="gemini-2.5-flash-native-audio-preview-09-2025",
            voice="Kore",
            # Optionally set a base system instruction; dynamic greeting will be used when generating reply.
            instructions="You are a helpful pizza delivery assistant."
        ),
        vad=silero.VAD.load(),
    )

    await session.start(agent=Assistant(), room=ctx.room)

    # Send the personalized first message to the user (text + voice if client supports it)
    await session.generate_reply(instructions=dynamic_instructions)

if __name__ == "__main__":
    agents.cli.run_app(
        agents.WorkerOptions(
            entrypoint_fnc=entrypoint,
            agent_name="test_agent",
        )
    )
If you want the agent to read user data from a different source (HTTP query params, JWT, or another room metadata key), tell me which source and the metadata keys you have and I’ll adjust the snippet.
Thanks — I looked through your logs and code. Quick summary: the LLM generates the correct Georgian greeting (logs show the exact text), but TTS fails afterward (
Error in _tts_inference_task
). That means the spoken audio never arrives even though the message text is correct. What to try next (minimal changes): 1. Remove any external 
tts=
configuration so Google Realtime handles audio itself. 2. Ensure you call 
await session.start(...)
then 
await ctx.connect()
before calling 
await session.generate_reply(...)
. 3. Send the exact first message text via 
generate_reply(instructions=...)
(or 
user_input=...
) after connect. Small tested agent.py adjusted to your setup (only one file; uses Google Realtime for LLM+TTS and Silero VAD):
Copy code
import os
from dotenv import load_dotenv
import logging
from livekit import agents
from livekit.agents import Agent, AgentSession, JobContext
from livekit.plugins import google, silero

load_dotenv(".env.local")
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)

def get_user_from_room(ctx: JobContext):
    md = getattr(ctx.room, "metadata", {}) or {}
    return {
        "name": md.get("user_name") or os.getenv("DEFAULT_USER_NAME", "ლაშა"),
        "city": md.get("user_city") or os.getenv("DEFAULT_USER_CITY", "Tbilisi"),
    }

class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful pizza delivery assistant.")

    async def on_enter(self):
        return

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    user = get_user_from_room(ctx)
    greeting = (
        f"გამარჯობა {user['name']}! მე ვარ პიცის მიტანის AI ასისტენტი. რით შემიძლია დაგეხმაროთ?"
    )

    session = AgentSession(
        llm=google.beta.realtime.RealtimeModel(
            model="gemini-2.5-flash-native-audio-preview-09-2025",
            voice="Kore",  # Google will handle synthesis
        ),
        vad=silero.VAD.load(),
    )

    await session.start(agent=Assistant(), room=ctx.room)

    # Important: call generate_reply after start() and after ctx.connect()
    await session.generate_reply(instructions=greeting)

if __name__ == "__main__":
    agents.cli.run_app(
        agents.WorkerOptions(
            entrypoint_fnc=entrypoint,
            agent_name="test_agent",
        )
    )
If that still logs 
_tts_inference_task
errors: • Tell me whether you are explicitly configuring any external TTS provider (Cartesia, ElevenLabs, etc.). If so, either remove it or share the TTS config/voice ID. • Share the exact TTS error stack (if different) and the language you expect (Georgian = 
ka
) so I can suggest a compatible TTS fallback. Try the file above and tell me the new logs or whether you hear the correct spoken greeting. Sources: Gemini Realtime Agent with Live Vision | LiveKit docs
2 Views
Open in Slack
PreviousNext
Powered by