dynamic variables in agent instructions

Thanks for reaching out, Saurabh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

agent is greeting multiple times

Thanks — I’m sorry you’re seeing the agent greet multiple times. I can help you diagnose and stop it. I’ll first list the most common causes and quick fixes, then ask a few targeted questions so I can give a precise solution. Why an agent may greet multiple times (common causes) • The greeting is triggered in an

on_enter

hook and that hook runs each time the agent becomes active (for example after a handoff or rejoin). • Workflows or handoffs cause the agent to be re-activated, which will re-run

on_enter

. • An

on_away

→ return flow triggers the greeting when the user comes back from inactivity. • Language switching or other runtime updates call the same greeting function repeatedly (example:

_switch_language

calling

await self.session.say(self.greetings[language_code])

each time). • No guard is in place to prevent repeated greeting calls (no flag/counter to suppress subsequent greetings). Quick fixes you can try now • Add a one-time flag or counter (e.g.,

self.greeting_played = True

) and check it in

on_enter

before sending the greeting. This ensures the greeting runs only once. • If you need a greeting per agent activation but not per rejoin, set the flag scoped to the session lifecycle (reset only when you intentionally want a new greeting). • For handoffs: only call the greeting when the handoff target changes in a way you expect (compare previous/next agent) rather than on every

on_enter

. • For

on_away

flows: add a minimum time or a boolean check to avoid greeting for brief returns. • For language switches or other updates: avoid calling the greeting directly from the language-change handler; instead, only announce the language change once or use a different short confirmation message. Targeted questions — this will help me give exact code or steps 1. Which hook is sending the greeting? (e.g.,

on_enter

,

on_away

, a language-change handler, or a custom event) 2. Are you using handoffs/workflows that re-activate the agent during a session? 3. Do you want the greeting exactly once per session, once per agent activation, or once per rejoin? 4. Are you building the agent programmatically (Python/SDK) or using the Agent Builder UI? 5. If programmatic, can you share the small snippet around where you call

session.say(...)

or

update_instructions()

(no secret keys)? If you tell me which hook or paste the relevant code snippet, I’ll provide a short code example showing exactly where to add a guard (flag/counter) and how to scope it correctly for your desired behavior. Sources: Pipeline nodes and hooks | LiveKit Documentation | How to implement call greeting retries in voice agents | LiveKit | ElevenLabs Change Language | LiveKit Documentation

livekit.greet ??

Thanks — I see you’re asking about

livekit.greet

and why your agent is greeting multiple times. I’ll keep this short and practical. Quick answer • There is no built-in

livekit.greet

function. Greetings are typically implemented by calling

session.generate_reply()

or

session.say()

(often from

on_enter()

or similar hooks). If your agent greets multiple times, it’s because whatever code calls that greeting is being run more than once. Why this happens (most common causes) •

on_enter()

runs each time the agent becomes active (handoffs, re-joins, or re-activation will retrigger it). •

on_away

→ return flows or reconnections can trigger a greeting when the user comes back. • Language-change handlers or other runtime updates call the same greeting function repeatedly. • No guard/flag exists to prevent the greeting from running again. What you can do right now (recommended fixes) • Add a one-time guard flag on the agent/session to stop repeated greetings: ◦ Example pattern:

async def on_enter(self) -> None:
        if not getattr(self, "_greeting_played", False):
            await self.session.generate_reply(instructions="Hello—welcome! How can I help?")
            self._greeting_played = True

• Scope the flag to the lifetime you want: ◦ Once per session: keep the flag until the session ends. ◦ Once per activation: reset it only when you intentionally want a new greeting. • For handoffs: only greet when the new agent is different from the previous one (compare agent IDs) or when a specific handoff reason occurs. • For language changes: avoid calling the full greeting; use a short confirmation instead (or check a flag before speaking the full greeting). If you want, I can: • Show the exact place to add this guard if you paste the small snippet where you call

generate_reply()

/

session.say()

or your

on_enter()

handler. • Provide a version scoped to handoffs or to

on_away

logic if that’s the flow you’re using. Which of these fits your situation? Are you using

on_enter

,

on_away

, a language-switch handler, or an Agent Builder configuration? Sources: Build a multilingual voice agent that automatically switches languages | Li... | Agent speech and audio | LiveKit Documentation | How to implement call greeting retries in voice agents | LiveKit | Agents and handoffs | LiveKit Documentation | Voice AI quickstart | LiveKit Documentation

agent is speaking very long sentences and not spotting on user interpption

Thanks — I hear you: the agent is speaking long sentences and not letting users interrupt. I can help you fix that. Below are focused actions you can take and a few quick questions so I can give an exact code snippet for your setup. What to do now (quick, high-impact fixes) • Shorten the agent’s outputs via instructions: ◦ Add a clear Output Rules section in the agent instructions like: ▪︎ "You are interacting via voice. Keep replies to 1–3 short sentences. Ask one question at a time. Do not use lists, code, or long explanations." ◦ This forces the LLM to produce speakable, concise text. • Ensure interruptions are enabled when generating speech: ◦ When using

session.say()

or

session.generate_reply()

, set

allow_interruptions=True

so user speech can cut in. • Use a guard to break long multi-part replies: ◦ Instead of one long monologue, structure the agent to produce stepwise replies and wait for user input between steps. Improve interruption detection (tuning) • Tune VAD parameters to detect short user interjections: ◦ Lower

min_speech_duration

and

min_silence_duration

to ~0.1s for more sensitive detection. • Use a turn-detection model for semantic detection of intent to speak: ◦ Add a Multilingual turn detector if brief interruptions are missed by raw VAD. • Adjust false-interruption handling: ◦ Increase

false_interruption_timeout

slightly so brief noises don’t always resume speech incorrectly. Practical code patterns (conceptual examples) 1) Prompt rules (put into your agent instructions):

Output rules:
- Keep replies to 1-3 short sentences.
- Ask only one question at a time.
- Avoid long monologues; stop speaking when the user interrupts.
- Plain text only; no lists, tables, or code blocks.

2) Generate interruptible speech:

await session.generate_reply(
    instructions="Greet briefly and ask how I can help. Keep to one sentence.",
    allow_interruptions=True
)

3) Use a greeting/response guard to avoid long repeats:

async def on_enter(self) -> None:
    if not getattr(self, "_has_spoken_initial", False):
        await self.session.generate_reply(instructions="Hello. How can I help?", allow_interruptions=True)
        self._has_spoken_initial = True

4) Tuning VAD (conceptual):

vad = silero.VAD.load(
    min_speech_duration=0.1,
    min_silence_duration=0.1,
    activation_threshold=0.5,
    prefix_padding_duration=0.1,
)
session = AgentSession(vad=vad, allow_interruptions=True)

Questions so I can give exact code/config for your case • Are you using the Agent Builder UI or programmatic agents (Python/SDK)? • Which API do you call to speak now —

session.say()

or

session.generate_reply()

? • Do you already have a VAD/turn-detection plugin configured? If so, which one and what are your current VAD params? • Do you want the agent to always stop immediately on any user audio, or only when the user speaks a minimum length (to avoid false triggers)? If you share which environment (Agent Builder vs Python SDK) and the small snippet where you call

generate_reply()

/

session.say()

(no secrets), I’ll give a ready-to-paste snippet that enforces brief speech and robust interruption handling for your exact flow. Sources: Prompting guide | LiveKit Documentation | Agent speech and audio | LiveKit Documentation | VAD and turn detection configuration guide | LiveKit | Turns overview | LiveKit Documentation | Tool definition and use | LiveKit Documentation

preamtive

Thanks — I want to make sure I understand what you mean by "preamtive." Do you mean one of these? • "preemptive" — the agent should preempt or interrupt itself (or proactively speak) before the user? • "preemptive interruptions" — the agent should allow or trigger early interruptions when it expects the user to speak? • "preemptive prompts" — the agent should give short, proactive prompts to guide the user? • Something else (please spell it out or give a short example)? Quick note: if you mean the agent should stop speaking as soon as a user interrupts (preemptive interruption handling), the actions I suggested earlier will help: set

allow_interruptions=True

, tune VAD/turn detection, and enforce short responses in the prompt instructions. Tell me which of the bullet options matches what you meant (or rephrase), and I’ll give a precise snippet and configuration for that behavior.

preemptive interruptions and preemptive prompts

Great — thanks for clarifying. Below I’ll show practical, actionable ways to implement both preemptive interruptions (make the agent stop quickly when the user speaks) and preemptive prompts (short proactive prompts that guide the user), with options for Agent Builder and programmatic (Python/SDK) setups. Short summary • Preemptive interruptions: enable interruptible speech, tune VAD/turn detection, set sensible time thresholds, and use short TTS segments so the agent can be cut off quickly. • Preemptive prompts: make the agent speak short, proactive prompts (1–2 sentences), then pause or wait for user input; use flags or small step flows to avoid repeating prompts. Preemptive interruptions — concrete steps 1) Enable interruptible speech • When you speak, set allow_interruptions=True:

await session.generate_reply(
  instructions="Briefly say the next step and then wait for the user.",
  allow_interruptions=True
)

2) Tune VAD and turn detection for quick response • Use a VAD config with low min durations to detect short interjections:

vad = silero.VAD.load(
  min_speech_duration=0.1,
  min_silence_duration=0.1,
  activation_threshold=0.5,
  prefix_padding_duration=0.05,
)
session = AgentSession(vad=vad, allow_interruptions=True)

• Add a turn-detection model for semantic intent detection if short interruptions are missed:

session = AgentSession(turn_detection=MultilingualModel(), allow_interruptions=True)

3) Shorten TTS chunks and avoid long monologues • Break long content into multiple short replies and only proceed after checking for user input. • Example pattern:

# Step 1 — say short part and wait for user
handle = await session.generate_reply(instructions="First step: press 1 for billing. Say 'repeat' to hear again.", allow_interruptions=True)
# Optionally await handle with interruption-aware logic or set a short timeout

4) Handle false interruptions gracefully • Configure false interruption handling to avoid resuming immediately after noise:

session = AgentSession(false_interruption_timeout=1.5, resume_false_interruption=True)

Preemptive prompts — concrete steps 1) Make prompts short and explicit in instructions • Add to instructions: ◦ "Ask one short question at a time. After each prompt, wait for the user's reply." • Example instruction snippet for Agent Builder: ◦ "Prompt the user with one sentence, then pause and listen. Keep it conversational and brief." 2) Use a proactive prompt pattern (programmatic example)

# Proactive short prompt
await session.generate_reply(
  instructions="Would you like help with billing or technical support? Please say 'billing' or 'support'.",
  allow_interruptions=True
)
# Then listen for user intent; if none, you can reprompt once

3) Use flags/counters to avoid repeating prompts • Track if you already prompted:

if not getattr(self, "_asked_topic", False):
    await session.generate_reply(instructions="Do you want billing or support?", allow_interruptions=True)
    self._asked_topic = True

Combined flow: preemptive prompts + interruptions • Prompt short, listen, and if interrupted, immediately process the partial user speech. • If not interrupted, optionally reprompt once after a short silence. • Example flow: a. Generate a one-sentence prompt with

allow_interruptions=True

. b. Start listening for speech; use VAD/turn detector. c. On interruption, immediately transcribe/process and cancel remaining TTS. d. If no response in N seconds, reprompt once (short) and then escalate or end. Agent Builder (no-code) guidance • Use Job metadata to pass any context that makes prompts targeted. • In the Advanced instructions, enforce output rules (1–2 sentences, wait for user). • Builder-generated greetings/prompts should include a line like: "Pause and listen after each prompt. Allow interruptions." Best-practice instruction text to copy/paste • Output rules (add to your agent instructions): ◦ "You are speaking via voice. Keep responses to 1–2 short sentences. Ask one question at a time. Stop speaking immediately if the user interrupts. After each prompt, pause and wait for the user's response." Diagnostics questions so I can give ready-to-paste code • Are you using Agent Builder or Python SDK? • Do you currently use silero VAD or a turn detection plugin? If so, what are the current settings? • Should prompts be repeated automatically if the user is silent, and how many retries are acceptable? If you tell me which environment (Agent Builder vs Python SDK) and share the short snippet where you call speech (or your current Agent Builder prompt text), I’ll provide an exact ready-to-paste configuration and code snippet tuned for preemptive interruptions and prompts.

vad = silero.VAD.load(
  min_speech_duration=0.1,
  min_silence_duration=0.1,
  activation_threshold=0.5,
  prefix_padding_duration=0.05,
)
session = AgentSession(vad=vad, allow_interruptions=True)

should this be in prewarm ?

Great question — thanks for sharing the snippet. Short answer: yes, placing VAD setup in your prewarm (or another early initialization phase) is recommended so the detector is ready before the agent starts speaking or listening. Below are why, how, and practical options. Why initialize VAD in prewarm • Performance: Loading and configuring the VAD once during prewarm avoids startup latency when a session starts, preventing missed early interruptions. • Resource reuse: A single initialized VAD instance can be reused across sessions or rooms if that fits your architecture, reducing memory/CPU overhead. • Deterministic behavior: Initializing VAD early ensures consistent parameters (sensitivity, padding) are applied the moment audio arrives. Recommended placements • Prewarm (best when supported): Initialize the VAD and any turn-detection models in your service’s prewarm/startup hook so they’re ready for incoming sessions. • Session creation (acceptable): If you need per-session customization (different thresholds per tenant/room), create the VAD during the AgentSession creation but still as early as possible—before calling

session.start()

or before any initial TTS. • Lazy-on-first-use (avoid if you need low-latency interrupts): Only if you cannot prewarm or create at session start; this risks missing very early user speech. Example patterns 1) Prewarm/global/shared instance (recommended)

# Run once at process startup / prewarm
vad = silero.VAD.load(
  min_speech_duration=0.1,
  min_silence_duration=0.1,
  activation_threshold=0.5,
  prefix_padding_duration=0.05,
)

turn_detector = MultilingualModel()  # optional

# Reuse for new sessions
def create_session(...):
    session = AgentSession(vad=vad, turn_detection=turn_detector, allow_interruptions=True)
    return session

2) Per-session initialization (when you need per-session tuning)

def create_session(custom_vad_params=None):
    params = custom_vad_params or {
      "min_speech_duration": 0.1,
      "min_silence_duration": 0.1,
      "activation_threshold": 0.5,
      "prefix_padding_duration": 0.05,
    }
    vad = silero.VAD.load(**params)
    session = AgentSession(vad=vad, allow_interruptions=True)
    return session

Notes and best practices • If you reuse a VAD instance across sessions, ensure the implementation is thread-safe or you create lightweight per-session wrappers if the underlying VAD mutates state. • Prewarm turn-detection models too if you use them — they also benefit from early loading to avoid latency. • Test with real calls: lowering durations increases sensitivity but can increase false positives in noisy environments. Tune

activation_threshold

and

min_*

values based on real-world audio. • If you expect multi-tenant differences, prewarm a default VAD and override per-session only when necessary. If you want, tell me: • Do you plan to reuse one VAD across sessions or create one per session? • Are you running agents in a multithreaded worker where thread-safety matters? I’ll give a ready-to-paste prewarm pattern for your exact deployment (single-process, multi-worker, or serverless).

allow_interruptions=True,
min_interruption_duration=0.3,
min_interruption_words=0,
min_endpointing_delay=0.5,
max_endpointing_delay=3.0,
# Handle false interruptions (e.g. background noise)
false_interruption_timeout=1.5,
resume_false_interruption=True,

fine tune for natural conversation

Thanks — you’ve shared this interruption config and asked to fine-tune it for natural conversation. Below I’ll explain what each parameter does, recommend tuned values for a natural, responsive flow, and give a ready-to-paste configuration plus notes for testing and adjustments. Your config (for reference)

allow_interruptions=True,
min_interruption_duration=0.3,
min_interruption_words=0,
min_endpointing_delay=0.5,
max_endpointing_delay=3.0,
false_interruption_timeout=1.5,
resume_false_interruption=True,

What each parameter controls (short) • allow_interruptions: lets users cut off TTS/agent speech. • min_interruption_duration: minimum continuous audio length from user before it’s considered an interruption. • min_interruption_words: minimum transcribed words required to treat it as a real interruption (helps filter noise). • min_endpointing_delay / max_endpointing_delay: the silence window used to decide when the user finished speaking (endpointing); max allows the system to wait longer if the turn detector is unsure. • false_interruption_timeout: time to wait after an interruption if no transcription appears — used to treat the event as false and resume agent speech. • resume_false_interruption: whether agent resumes speaking after a false interruption. Recommended tuned settings for natural conversation • allow_interruptions=True ◦ Keep enabled for natural turn-taking. • min_interruption_duration=0.25 ◦ Slightly more responsive than 0.3; captures quick interjections while avoiding extremely brief noises. • min_interruption_words=1 ◦ Requires at least one transcribed word to count as a user intented interruption (reduces false triggers from non-speech noise). Use 0 only if your environment is very clean and you need immediate VAD-only cuts. • min_endpointing_delay=0.25 ◦ Shorten to make endpointing faster for quick replies. • max_endpointing_delay=2.5 ◦ Keep a reasonable upper bound so longer thoughtful responses are still captured but you don’t wait too long. • false_interruption_timeout=1.2 ◦ Slightly shorter than 1.5 to reduce resume latency after a noise false positive, while still allowing brief pauses for transcription to start. • resume_false_interruption=True ◦ Keep true so the agent resumes when interruptions were false positives. Tuned config (ready to paste)

allow_interruptions=True,
min_interruption_duration=0.25,
min_interruption_words=1,
min_endpointing_delay=0.25,
max_endpointing_delay=2.5,
false_interruption_timeout=1.2,
resume_false_interruption=True,

Additional recommendations to improve naturalness • Use a turn-detection model (e.g., MultilingualModel) in addition to VAD. This greatly reduces false interruptions and improves semantic detection of user intent. • Prewarm VAD and turn-detector models so they are ready before any early speech. • Keep agent replies short (1–2 sentences) in the prompt instructions so there’s more opportunity for user turn-taking. • Use preemptive generation (if available) to reduce latency between user turn end and agent response. • Monitor metrics in real calls and tune incrementally: ◦ Increase min_interruption_words if false positives persist. ◦ Increase min_interruption_duration if noisy environments create spurious cuts. ◦ Adjust false_interruption_timeout upwards if transcriptions often arrive slightly late. Testing plan (quick) 1. Test in a quiet environment with multiple quick interjections (e.g., "wait", "yes", "stop") to verify responsiveness. 2. Test in a noisy environment and increase min_interruption_words or min_interruption_duration if false triggers increase. 3. Log interruption events and whether a transcription followed — tune false_interruption_timeout based on observed transcription delays. 4. If interruptions are still missed, enable/adjust turn-detection and consider lowering min_endpointing_delay slightly. If you want, I can: • Give you a combined snippet that includes VAD load, MultilingualModel setup, and AgentSession construction with these tuned values. • Help you pick starting values based on the noise profile of your target environment (e.g., call center vs. quiet home). Which environment are you testing in? Sources: Turns overview | LiveKit Documentation | Uninterruptable Agent | LiveKit Documentation | examples/voice_agents | livekit/livekit_composite | DeepWiki | How to match the latency of the homepage agent | LiveKit | Improving voice AI's turn detection with transformers