https://livekit.io logo
Join Slack
Powered by
I want to use llm generated greetings and differen...
# ask-ai
r
I want to use llm generated greetings and different behaviours if user speaks first. upon silence agent should greet
t
Thanks for reaching out, Saurabh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great idea and totally doable with LiveKit Agents. I’m responding to your request to have LLM-generated greetings and different agent behaviors depending on whether the user speaks first, and to have the agent greet if the user is silent. Before I give a concise implementation plan, a quick question so I tailor the example correctly: • Are you using a specific LLM or provider (e.g., OpenAI, Gemini, a local model)? • Which runtime are you integrating with LiveKit Agents (Python/async, Node, or another SDK)? If you want a general approach now, follow these steps. I’ll keep it focused and you can ask for code in your chosen language afterward. 1) Detect silence (user-initiated vs agent-initiated) • Configure 
user_away_timeout
on the AgentSession to mark the user as “away” after N seconds of silence. This lets the system know when nobody has spoken. • Subscribe to the 
user_state_changed
event to be notified when the user becomes 
away
(silent) or 
speaking
(user spoke first). 2) Branch behavior depending on who speaks first • If 
user_state_changed
becomes 
speaking
before your timeout fires, treat this as “user spoke first.” In that branch, instruct the LLM to respond to the user’s message and follow the normal conversational flow. • If the user becomes 
away
(no speech detected) after the timeout, treat this as “user silent.” In that branch, have the agent proactively generate a greeting and follow-up prompts. 3) Use LLM instructions to generate context-aware greetings and behaviors • When agent should initiate: call the LLM with an instruction like: ◦ "You are initiating the conversation. Greet warmly, introduce yourself, and ask an open question to invite the user to speak." • When user initiated: call the LLM with an instruction like: ◦ "User started the conversation. Respond to their last utterance warmly and offer help relevant to what they said." 4) Optional: retry and end-of-call behavior • Keep a small retry counter for greeting attempts. Example policy: ◦ Repeat greeting up to 2 times (with slightly different wording). ◦ After retries are exhausted and no response, say a short goodbye and end or hand off the session. • This avoids repeated identical greetings and makes behavior feel natural. 5) Practical flow (high level) 1. Start AgentSession with 
user_away_timeout = X
seconds (e.g., 5s). 2. Listen for `user_state_changed`: ◦ If `speaking`: Cancel any pending agent-initiated greeting and route turn to LLM to respond. ◦ If `away`: Wait 0–100ms to ensure it's not a short pause, then call LLM to generate the greeting and play it via TTS. 1. After agent greeting: ◦ Start a short follow-up timer to detect response. If no response, retry or end per your retry policy. 6) Example instruction variations to send to the LLM • Agent-initiated greeting: “Greet the user in a friendly tone, introduce yourself as the assistant, and ask how you can help. Keep it concise and invite them to speak.” • If user started: “Read the user’s last utterance and respond directly. Be helpful and concise; ask a relevant follow-up question.” What I can provide next • A short runnable snippet in the language you’re using (Python or Node) showing: ◦ setting 
user_away_timeout
◦ handling 
user_state_changed
◦ calling 
generate_reply(instructions=...)
◦ retry logic for silence • Guidance on instruction design for the LLM to shape tone and behavior. Tell me which LLM and SDK/lang you prefer and I’ll send a focused code example and exact instruction templates. Sources: How to detect and handle silence during calls | LiveKit | Agent speech and audio | LiveKit Documentation | How to implement call greeting retries in voice agents | LiveKit
r
how to pass dynamic variables to instructions and greetings ?
t
Thanks — I’m responding to your question about how to pass dynamic variables into LLM instructions and greetings so your agent can personalize messages at runtime. Here are practical ways to inject dynamic values, when to use each, and concise examples you can adapt. 1) Format instructions before sending (client-side interpolation) • When to use: simple, immediate personalization when you control the prompt text. • How: build the instruction string with your variables and pass it to the LLM call (or to 
generate_reply()
). • Example (Python-style):
Copy code
instructions = f"Greet the user by name. Hello {user_name}! Ask how I can help."
await session.generate_reply(instructions=instructions)
• Notes: This is straightforward and works anywhere you can call the LLM/generate_reply. 2) Use context variables in your Agent class (session startup) • When to use: you want the agent to have persistent context info (user name, account tier, locale) for the whole session. • How: format the agent 
instructions
at initialization with a context dictionary. • Example:
Copy code
class ContextAgent(Agent):
    def __init__(self, context_vars=None):
        instructions = """
You are a helpful assistant. The user's name is {name}. They prefer {language}.
"""
        if context_vars:
            instructions = instructions.format(**context_vars)
        super().__init__(instructions=instructions)

# start session
ctx = {"name": "Alex", "language": "French"}
await session.start(agent=ContextAgent(context_vars=ctx), room=room)
• Notes: Keeps variables available to all LLM turns without repeating them each call. 3) Use job/agent builder metadata (templating with {{metadata.*}}) • When to use: using LiveKit Agent Builder or dispatched jobs where metadata is supplied at job creation. • How: include metadata JSON when dispatching, then reference with 
{{metadata.var_name}}
inside builder instructions or greeting templates. • Example: ◦ Dispatch metadata: 
{ "user_name": "Priya", "company": "AcmeCo" }
◦ In builder greeting: "Hello {{metadata.user_name}}, welcome to {{metadata.company}}!" • Notes: Great for no-code/customizable behaviors and for passing values from your backend into the agent config. 4) Read room or participant attributes at runtime • When to use: you store data on the room or participant (for example from a prior flow) and want to read it during the session. • How: fetch attributes and inject into your instruction before generating a reply. • Example:
Copy code
user_name = ctx.room.local_participant.attributes.get("user_name")
await session.generate_reply(instructions=f"Hello {user_name}, how can I help today?")
• Notes: Useful for user-specific settings kept in LiveKit participant metadata. 5) Combine dynamic runtime data with conditional instructions • When to use: behavior should change depending on variable values (e.g., premium vs free user). • How: build different instruction templates and choose one based on variables. • Example:
Copy code
if account_tier == "premium":
    instructions = f"Address {user_name} with priority support tone..."
else:
    instructions = f"Hello {user_name}, here's how I can help with our standard support..."
await session.generate_reply(instructions=instructions)
6) Best practices and tips • Sanitize/escape variables before interpolation to avoid unintended prompt structure or injection. • Keep variable injection concise in the instruction; supply longer context via structured system messages or context variables. • Use placeholders consistently (e.g., 
{user_name}
or 
{{metadata.user_name}}
) so templates are easy to manage. • For multi-turn personalization, prefer context variables or agent-level instructions so you don’t need to re-inject on every turn. • If repeating greetings, vary phrasing slightly on retries to avoid sounding robotic. If you tell me: • which SDK/language you’re using (Python, Node), and • where your dynamic data is stored (backend, room attributes, job metadata), I’ll provide a short runnable snippet and a couple of ready-to-use instruction templates (greeting, follow-up, retry) tailored to your setup. Sources: Context Variables | LiveKit Documentation | Agent Builder | LiveKit Documentation | How to implement call greeting retries in voice agents | LiveKit | External data and RAG | LiveKit Documentation
r
llm generated greeting
t
Thanks — I’m responding to your request for an LLM-generated greeting. Below are concise, practical ways to generate a dynamic, context-aware greeting (including silence handling), plus ready-to-use instruction templates and short code snippets you can drop into your LiveKit Agent flow. Quick question before examples: • Which SDK/language do you want the snippet in (Python or Node)? Tell me and I’ll provide a runnable example tailored to your stack. 1) Core idea (how it works) • Use 
session.generate_reply()
to ask the LLM to produce the greeting text. • Inject dynamic variables (user name, time_of_day, account_tier, etc.) into the instruction before calling 
generate_reply()
(or use agent/job metadata templating). • Use 
user_away_timeout
+ 
user_state_changed
to detect silence vs user-first speech and branch behavior. 2) Short instruction templates (copy/paste into 
generate_reply()
) • Agent-initiated greeting (warm, short): "Greet the user by name, introduce yourself briefly, and ask an open question to invite them to speak. Keep it friendly and concise. Example: 'Hi {user_name}, I’m [Assistant]. How can I help today?'" • User-initiated response (acknowledge + help): "The user started. Acknowledge their last utterance, respond helpfully, and ask one clarifying question if needed. Be concise and polite." • Follow-up after silence (retry once): "The user didn’t respond to the first greeting. Politely check if they’re there, using a slightly different phrasing. Keep it brief." • Final goodbye after retries exhausted: "No response after repeated attempts. Say a short goodbye and indicate the user can reconnect if they need help." 3) Example: minimal Python pattern (adapt to your session) • Build the greeting instruction with variables, then call the LLM.
Copy code
user_name = metadata.get("user_name", "there")
time_of_day = metadata.get("time_of_day", "")

greeting_instr = (
    f"Greet the user by name. Example: 'Hi {user_name}, good {time_of_day}! "
    'I’m your assistant — how can I help today? Keep it friendly and concise.'
)

# Generate & play greeting
await session.generate_reply(instructions=greeting_instr)
4) Silence handling & retries (pseudo-flow) 1. Start session with 
user_away_timeout
(e.g., 5s). 2. On `user_state_changed`: ◦ If `speaking`: cancel any pending greeting tasks and route the user's speech to the LLM. ◦ If `away`: run a follow-up task that: ▪︎ Calls 
generate_reply()
with the follow-up instruction. ▪︎ Increments a retry counter; if retries exhausted, call the goodbye instruction and end/hand off the session. 5) Personalization and dynamic variables • Options to supply variables: ◦ Format the instruction string client-side (easy and immediate). ◦ Use Agent/Session context variables at initialization (persist across turns). ◦ Use job/agent metadata templating (
{{metadata.var}}
) if using Agent Builder. ◦ Read participant/room attributes at runtime and inject into instruction. • Always sanitize variables before injecting to avoid accidental prompt issues. 6) Tone & phrasing tips for the LLM • Ask the LLM to be concise (limit to 1–2 sentences) to reduce TTS latency. • Provide examples in the instruction so the LLM matches style. • Vary phrasing between retries to avoid repetition. If you tell me: • Python or Node example preferred, and • Where your dynamic data lives (job metadata, backend, room attributes), I’ll send a runnable snippet that includes: • initial LLM-generated greeting, • silence detection and retry logic, • one or two polished instruction templates ready for production. Sources: How to implement call greeting retries in voice agents | LiveKit | ElevenLabs Change Language | LiveKit Documentation | Testing and evaluation | LiveKit Documentation
2 Views
Open in Slack
PreviousNext
Powered by