The user wants me to do proper research into LiveKit's SDK and guidelines before making any changes. Let me research thoroughly: 1. How does the LiveKit Python agents SDK handle tool_choice with the Realtime API? 2. What's the recommended pattern for greeting + tool calls? 3. How does generate_reply actually work with tool chains? Let me look at the actual SDK source code and the examples to understand the right pattern. The key research finding was that generate_reply supports per-call tool_choice parameter. Let me also look at how the existing re-entry path works (it uses tool_choice="required" successfully). Now I need to dig into the SDK source for generate_reply and tool execution, check out the form_agent example, and see how other agents in the livekit-examples repo handle this pattern to find the right approach. ⏺️ Let me do proper research first — SDK source, LiveKit examples, and the actual tool execution pipeline.

Thanks for reaching out, Samuel. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

How to have metadata sent for the given user query: dynamic LLM chips generation on the fly

Thanks — this is a good, specific use case. I can give you exact patterns and code once I know a few details about your setup and where you want the metadata to be consumed. A few quick questions to narrow this down: 1. Which SDK are you using? (LiveKit Agents Python, LiveKit Agents Node, or the Realtime client SDK in a browser/native app?) 2. Do you want the metadata sent to the LLM at generation time, to server-side agent/tool functions, or to the client UI that renders the chips? 3. Is the metadata scoped per-session (room/job), per-participant, or per-query (different every user utterance)? 4. If per-query, how is the query created? (user types in client UI, server calls

generate_reply()

, or a Job dispatch triggers it?) 5. Where are the chips rendered? (client-side UI reads structured result, or server returns a UI payload?) 6. Can you share an example metadata shape and the expected chip output format? (e.g., metadata:

{ user_id, preferences, available_actions }

→ chips:

[{"label":"Pay","action":"pay_url"}, ...]

) What I can already confirm and recommend based on LiveKit Agents functionality: • You can pass metadata through job metadata (JSON string) when dispatching an agent; access it in the agent entry via

ctx.job.metadata

(parse JSON). This is ideal for per-job/session-level user context. • Room metadata (

ctx.room.metadata

) and participant attributes are also available and useful for persistent session or participant-level context. • For per-query dynamic metadata, attach it at generation time — either: ◦ Include it in the

instructions

or

user_input

you pass into

generate_reply()

so the LLM receives it as part of the prompt, or ◦ Pass it as structured tool arguments to a function tool (decorated

@function_tool

) that returns structured data the client can use to render chips. • To generate chips dynamically and reliably for client rendering, return structured JSON from the agent/tool rather than relying on free-form text. A function tool that returns a JSON array is the cleanest pattern; the client can then map that array to chip UI elements. If you tell me the SDK and which scope (job/room/query) you need, I’ll provide a concrete code example for that SDK showing: • How to attach the metadata, • How to surface it to the model or tool, • How to return structured chips that the client can render. Sources: Job lifecycle | LiveKit Documentation

i use python SDK and JS SDK

Thanks — great details and a clear goal. I’ll show a concise, practical approach for sending metadata with user queries (both Python and JS SDKs) and how to use that metadata to generate dynamic LLM “chips” on the fly. I’ll cover the recommended metadata scopes, how to attach metadata, how the agent can consume it, and a reliable pattern for returning structured chips to the client (RPC + function tools). Quick summary • Use job metadata for per-dispatch/session context, participant attributes for dynamic per-session updates, and room metadata for shared context. • For per-query metadata (different every user utterance), include it when calling

generate_reply()

or send it to the agent via participant attributes or an RPC trigger. • Return structured JSON from a function tool or via an RPC call so the frontend can render chips reliably. • Pattern: Agent detects intent → calls a function tool that creates structured chips → tool sends chips to the client via RPC → client renders chips and sends back the chosen action. 1) Recommended metadata scopes and when to use them • Job metadata (set when dispatching an agent): good for initial context (user_id, account, preferences). • Participant attributes: good for live, mutable state (preferences changed mid-call, urgency flags). • Room metadata: shared room-level context (meeting id, mode). • Per-query metadata: attach at generation time (include in `instructions`/`user_input`) or send as structured payload via an RPC or direct call to the agent. 2) Pattern for per-query metadata and dynamic chip generation (high level) 1. Client sends user utterance + optional per-query metadata to server or directly sets participant attributes. 2. Agent receives the message / sees updated attributes. 3. Agent calls a function tool (or calls

generate_reply(tool_choice="required")

if LLM must consult a tool). 4. Tool returns structured JSON representing chips (labels + actions/IDs/urls). 5. Tool sends that JSON to the client via RPC. Client renders chips. 6. Client sends back user selection (as message, attribute change, or RPC), agent processes the selection. 3) Concrete examples A. Dispatch agent with job metadata (Python) • Use job metadata for initial context.

# When dispatching (server-side)
await lkapi.agent_dispatch.create_dispatch(
    api.CreateAgentDispatchRequest(
        agent_name="my-agent",
        room="my-room",
        metadata='{"user_id":"123", "initial_intent":"support"}'
    )
)

Agent reads metadata in entrypoint:

@server.rtc_session(agent_name="my-agent")
async def my_agent(ctx: JobContext):
    metadata = json.loads(ctx.job.metadata)  # string -> dict
    user_id = metadata.get("user_id")
    # use metadata in prompts or to tailor chips

B. Per-query metadata via generate_reply (Python) — include structured context with the call • Useful when each user query has different contextual flags.

await session.generate_reply(
    user_input="I want to book a meeting",
    instructions='Use this metadata: {"timezone":"PST","availability":["9:00","13:00"]}'
)

Alternatively, pass per-query metadata into a function tool rather than free-form text. C. Participant attributes (JS client) — dynamic updates mid-call • Update attributes client-side so the agent sees them and triggers chip generation.

await room.localParticipant.setAttributes({
  "user.intent": "schedule_meeting",
  "user.timezone": "PST",
  "user.pref": "video"
});

Agent listens for attribute changes and triggers chip generation:

# inside agent
@ctx.room.on("participant_attributes_changed")
async def on_attrs(attrs, participant):
    if participant.identity == ctx.room.local_participant.identity:
        # parse attrs and call tool or generate_reply
        await generate_chips_based_on_attrs(ctx, attrs)

D. Function tool that returns structured chips and sends them via RPC (Python) • This is the most reliable way to produce deterministic chips the UI can render.

@function_tool()
async def make_chips(context: RunContext, options: list[dict]):
    """
    options example: [{"label":"9 AM","value":"9:00"},{"label":"1 PM","value":"13:00"}]
    """
    # Send chips to frontend via RPC to a specific participant
    dest = list(context.room.remote_participants.keys())[0]  # choose client identity
    payload = json.dumps({"chips": options})
    await context.room.local_participant.perform_rpc(
        destination_identity=dest,
        method="showChips",
        payload=payload
    )
    return {"status": "sent", "count": len(options)}

JS client receives RPC and renders chips:

room.on("rpcReceived", (msg) => {
  if (msg.method === "showChips") {
    const data = JSON.parse(msg.payload);
    renderChips(data.chips); // update UI state
  }
});

E. Example end-to-end flow 1. Dispatch with job metadata (user profile). 2. Client or agent sets/updates participant attributes if per-query changes are needed. 3. User says: “Book something at 9”. 4. Agent decides to generate chips (call to

make_chips

), passing available options. 5. Frontend receives chips via RPC and displays buttons. 6. User clicks a chip; client sends either an RPC back or a standard message like “Select900”. 7. Agent processes the selection and confirms. 4) Notes on using

generate_reply()

vs function tools • Use

generate_reply()

when you want the LLM to generate natural language responses (optionally instructing it with metadata in

instructions

). • Use function tools (decorated with

@function_tool

) when you need deterministic structured output (JSON chips), side effects (RPC to UI), or to call external services. • You can combine them: require tools for certain steps by calling

generate_reply(tool_choice="required")

so the LLM must call the defined function tool that returns chips. 5) Guidance on structure and reliability • Return structured JSON (array of objects) for chips: label, id/value, optional metadata (icon, color, target_url). • Example chip format: { label: "9 AM", value: "2026-03-05T090000Z", meta: { type: "time" } } • Always send chips via RPC or a structured tool return — don’t rely on parsing free-form LLM text to build UI components. 6) Follow-ups I can help with • Provide a full working Python agent + minimal React/JS client example demonstrating RPC-driven chips. • Show how to attach per-query metadata into the LLM prompt cleanly (prompt templates). • Suggest JSON chip schema and how to handle click events securely in the client. Which would you like next: a full working example (agent + client), or a focused snippet showing how to attach per-query metadata into

generate_reply()

and return chips via a function tool? Sources: Job lifecycle | LiveKit Documentation | Handling asynchronous metadata updates in LiveKit | LiveKit | External data and RAG | LiveKit Documentation

See my Agent & User flow is like this. I need to generate on the fly dynamic chips. Agent: What type of equipment would you like to sell? (i've predefined list like: tractor, dozer, truck and so on) So chips has to be shown. User: Click Tractor Chips Agent: Ah, tractor. what is the make? dynamic chips is to be created on the fly. like top manufactures of tractor such as John Deere, JCB, Masey Ferguson and so on

Thanks — this flow is clear and a perfect fit for LiveKit’s real-time data messaging + agent tooling. Below is a compact, actionable plan and code examples (Python agent + JS client) you can drop into your app to generate chips on the fly exactly as you described. High-level flow (what you want) 1. Agent asks: “What type of equipment would you like to sell?” — agent sends initial chips (tractor, dozer, truck). 2. User clicks “Tractor” chip in the client. 3. Client sends that selection to the agent (as a data message or text). 4. Agent receives selection, computes dynamic options (top manufacturers for Tractor), and sends a new chips payload to the client. 5. Client renders the new chips; repeat as needed. Why this works • Use LiveKit data messages (topics) so the agent can push structured chip payloads to the client instantly. • Use a small, deterministic JSON schema for chips so the client can reliably render buttons and return a clear selection back to the agent. • Handle clicks as text messages or dedicated data messages so the agent treats them as normal user input and continues the flow. Recommended JSON chip schema • Example: { label: "John Deere", value: "john_deere", meta: { type: "manufacturer", equipment: "tractor" } } Minimal end-to-end example 1) Agent (Python) — send initial chips and respond to selection • Send chips using publish_data (topic "lk.chips") so client renders them. • Listen for incoming chat/data to detect chip selection, generate next chips, and send them. Python (agent) snippets: Send chips helper

import json

async def send_chips(session, dest_identity, prompt, options):
    payload = {"type":"chips", "prompt": prompt, "options": options}
    # publish to all participants (or use perform_rpc to a specific identity)
    await session.room.local_participant.publish_data(
        data=json.dumps(payload).encode(),
        reliable=True,
        topic="lk.chips"
    )

Initial greeting & chips

# After session.start / on agent entry
equipment_options = [
    {"label":"Tractor","value":"tractor"},
    {"label":"Dozer","value":"dozer"},
    {"label":"Truck","value":"truck"},
]
await send_chips(session, None, "What type of equipment would you like to sell?", equipment_options)

Handle incoming selection (chat/data)

# Example handler when you receive a chat message or data event containing selection
async def on_user_message(session, message_text):
    # message_text could be "select:tractor" or free text; parse accordingly
    if message_text.startswith("select:"):
        selected = message_text.split("select:")[1].strip()
        if selected == "tractor":
            # compute dynamic manufacturers for tractor
            manufacturers = [
                {"label":"John Deere","value":"john_deere"},
                {"label":"JCB","value":"jcb"},
                {"label":"Massey Ferguson","value":"massey_ferguson"}
            ]
            await send_chips(session, None, "Ah, tractor. What is the make?", manufacturers)
        else:
            # handle other equipment types similarly
            pass
    else:
        # fall back to regular NLP flow, or parse natural-language selection
        pass

Notes: • You can also call a function tool (decorated

@function_tool

) to generate chips if you want deterministic JSON from the agent/tool pipeline. • For targeted delivery to a single participant, use

perform_rpc(destination_identity=..., method=..., payload=...)

instead of

publish_data

. 2) Client (JS) — receive chips and send selection • Listen for data messages with topic "lk.chips". Render chips UI and on click send selection back to the agent. Client (JS/React) snippets: Listen and render

useEffect(() => {
  const handleData = (payload, participant, topic) => {
    if (topic === "lk.chips") {
      const message = JSON.parse(new TextDecoder().decode(payload));
      if (message.type === "chips") {
        setPrompt(message.prompt || "");
        setChips(message.options || []);
      }
    }
  };

  room.on("data_received", handleData);
  return () => room.off("data_received", handleData);
}, [room]);

Handle chip click (send selection)

const handleChipClick = async (chip) => {
  // Option A: send as text chat (agent will parse "select:VALUE")
  await room.localParticipant.sendText(`select:${chip.value}`, { topic: "lk.chat" });

  // Option B: send as data message (more structured)
  await room.localParticipant.publishData(JSON.stringify({
    type: "chip_click",
    value: chip.value,
    meta: chip.meta || {}
  }), { reliable: true, topic: "lk.chips.click" });

  // Clear chips UI if you want
  setChips([]);
};

3) Agent — receive structured click (preferred) • Prefer structured data ("lk.chips.click") so parsing is unambiguous. Listen for that topic and continue the same handler logic as above. Agent-side listening (conceptual) • If you publish clicks as

lk.chips.click

, have the agent parse that payload and call send_chips for the next step. Best practices and tips • Use structured JSON for chips; don’t rely on parsing LLM free-form text to build UI options. • Keep chip payloads small and deterministic: label, value, optional meta. • Prefer dedicated topics for chip payloads and chip clicks (e.g., "lk.chips" and "lk.chips.click") to avoid mixing with general chat. • If multiple clients exist in the room, target the right participant using

perform_rpc

or include the intended recipient identity in your payload. • If your chip generation relies on LLM reasoning, call

generate_reply(tool_choice="required")

or a function tool that returns JSON chips so the LLM is forced to call your deterministic tool for chip creation. • For accessibility, also allow keyboard input or voice fallback if the user doesn’t click chips. Would you like: • A full working repo-style example (Python agent + small React client) you can run locally, or • A compact PR-style patch for your existing agent showing exactly where to integrate the send_chips / receive handlers? Which do you prefer and I’ll prepare it next. Sources: client-sdk-swift | livekit/livekit_composite | DeepWiki | Client SDKs | livekit/livekit_composite | DeepWiki | examples/other | livekit/livekit_composite | DeepWiki