I'm building a LiveKit voice agent (Python Agents ...
# ask-ai
b
I'm building a LiveKit voice agent (Python Agents SDK) with a web chat UI. I want to show clickable option chips in the frontend alongside the agent's speech. Current setup: - Agent has a show_options(options: list[str]) function_tool that sends chips via RPC - For catalog questions, the LLM calls get_services → show_options → works fine - For dynamic questions (year, yes/no, condition) the LLM often skips show_options, even with prompt instructions. tool_choice="required" causes either infinite tool loops or suppresses speech entirely. What I want: After the LLM generates a response (speech + any tool calls), I want to reliably auto-send chips to the frontend based on what question the agent just asked — WITHOUT requiring the LLM to decide to call show_options. Specific questions: 1. Is there an AgentSession callback I can hook into AFTER the LLM finishes a turn (after speech text is available but before or during playout)? e.g. on_reply_generated, after_llm_cb, or similar? 2. In that callback, can I inspect the agent's response text, then programmatically call my RPC method (agent.displayUI) to push chips to the frontend? 3. What's the correct LiveKit Agents v1 Python pattern for this — intercepting agent responses to inject side-effects? I've seen references to before_llm_cb in the SDK. Is there an after equivalent? 4. Is there a nova-sonic / form_agent example that does something similar (post-processing LLM output to drive UI updates)? SDK version: livekit-agents 0.12.x (Agents v1 API) --- What you're looking for: a Python hook like after_llm_cb or an on_message event on AgentSession that fires after each turn with the text response, so we can analyze the speech text and auto-call show_options server-side — making chip display deterministic instead of LLM-dependent.
t
Thanks for reaching out, Samuel. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great, clear question. I’m responding to your four specific asks about reliably pushing option chips after each agent utterance (so the LLM doesn’t need to call show_options). Short answer • Use the AgentSession event conversation_item_added. It fires after each message (including the assistant’s completed reply), so you can inspect the final text and then call your RPC (e.g., agent.displayUI) to push chips. This makes the UI deterministic and server-controlled. What to hook into (Q1) • Use session.on("conversation_item_added"). This event is triggered when any conversation item (user or assistant) is added to the session. It runs after the agent’s reply text is available, so it’s the reliable post-generation hook you want. Can you inspect text and call RPC there? (Q2) • Yes. In that handler, filter items where role == "assistant", extract the text, run your logic to determine chip options, then call your RPC to the frontend (for example, via ctx.room.local_participant.rpc.invoke or whatever RPC mechanism you already use). That will push chips regardless of whether the LLM invoked show_options. Recommended pattern and example (Q3) • Pattern: subscribe to conversation_item_added → detect question type from assistant text → call RPC to display chips. • Key points: ◦ Do not rely on the LLM to call show_options for deterministic UI. ◦ Keep the detection logic server-side (simple heuristics or a small classifier). ◦ Optionally, disable or deprioritize the show_options tool in prompts to avoid tool loops. Minimal example (Python-like pseudocode)
Copy code
@session.on("conversation_item_added")
async def on_conversation_item(item: ChatItem):
    if item.role != "assistant":
        return

    text = item.text_content()
    options = detect_question_type(text)  # your heuristic/classifier
    if not options:
        return

    await ctx.room.local_participant.rpc.invoke(
        method="agent.displayUI",
        data={"type": "options", "options": options}
    )
detect_question_type suggestions • yes/no: if text contains phrases like "yes or no", leading question verbs, or ends with a yes/no prompt → return ["Yes","No"] • year/date: detect tokens like "what year", "when", or pattern matching for numbers → return plausible years/options • condition/choices: parse enumerations or known catalog keys when present Notes on tool_choice and avoiding loops • Setting tool_choice="required" leads to tool loops or suppressed speech as you observed. With the server-side event approach, you can: ◦ Keep show_options as an optional tool (for when the model intentionally wants to call it). ◦ Rely on conversation_item_added for deterministic chip pushes. ◦ Optionally remove or temporarily hide the show_options tool in the agent’s toolset if you want to fully prevent the model from calling it. Other hooks (when you might use them) • Speech playback callbacks (e.g., SpeechHandle callbacks) exist if you need to run something strictly after audio playout finishes (visual feedback tied to TTS end). But for getting the final text and pushing UI, conversation_item_added is simpler and more reliable. Examples & references (useful) • conversation_item_added is part of the AgentSession lifecycle and is the right post-generation hook for this use case. • You can combine this with small server-side state (a short state machine) to ensure the correct chip set for multi-step flows. If you’d like, I can: • Provide a complete working agent template (full file) with a robust detect_question_type implementation. • Show a version that also waits for TTS end before showing chips, if you need chips to appear exactly after audio completes. Which would you like next: the full template, or a focused detect_question_type implementation tuned to your catalog + dynamic questions? Sources: Agent session | LiveKit Documentation | Turns overview | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation
b
Thank you — conversation_item_added is exactly what we need. Important context before I implement: this is a UNIVERSAL, domain-agnostic voice agent. The same agent codebase serves multiple business tenants (agricultural equipment, real estate, healthcare, automotive, etc.) — each with their own system prompt loaded from a database at runtime. Current architecture: - Tenant provides a services_catalog (JSON) with domain-specific option groups e.g. { "equipment_type": {items: [...]}, "property_type": {items: [...]} } - Agent has get_services(group) tool → looks up options from services_catalog - Agent has show_options(labels) tool → sends chips to frontend via RPC - Chips that work today: catalog-backed (LLM calls get_services → show_options) - Chips that FAIL: dynamic/universal (yes/no, date ranges, scales like 1-10) because the LLM skips calling show_options even when prompted The over-engineering concern: adding a conversation_item_added handler with hardcoded keyword patterns for one domain (equipment) would break the universal design. Questions: Q1. In a domain-agnostic system, should show_options remain an LLM-callable tool (letting the LLM decide labels based on its domain knowledge from the system prompt), OR should we move to event-driven detection? Q2. If event-driven: how do we make chip detection domain-agnostic? Can the agent embed structured metadata in its speech (e.g. a special tag or JSON prefix) that the event handler parses to extract options — without the LLM needing a separate tool call? Example: LLM outputs "What year is it? [[chips:2024,2023,2022,Older]]" Event handler strips the tag, pushes chips, sends clean text to TTS. Q3. Is there a pattern in LiveKit Agents (structured output, tool_choice, or otherwise) where the LLM is forced to always emit chip metadata alongside its response — making it impossible to "forget"? Q4. What do production multi-tenant agents use for reliable UI updates that scale across domains without hardcoded domain logic? Goal: one clean system that works for any tenant/domain without per-domain keyword detection code.
t
I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!