https://livekit.io logo
Join Slack
Powered by
What's the difference between `generate_reply(user...
# ask-ai
b
What's the difference between 
generate_reply(user_data="...")
v.s. 
generate_reply(instructions="...")
in an AgentSession
✅ 1
t
Thanks for reaching out, Masayuki. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — you’re asking about the difference between calling 
generate_reply(user_data="...")
vs 
generate_reply(instructions="...")
in an AgentSession. I don’t have documentation that lists a 
user_data
parameter for `generate_reply`; the documented parameters I can see are 
user_input
and 
instructions
. Because 
user_data
isn’t confirmed, I need one quick clarification before giving a precise recommendation: • By 
user_data
, do you mean: ◦ A raw user message you want the agent to reply to (i.e., the user's latest text), or ◦ Structured/context data (JSON, profile, conversation metadata) that should be used as background context? What I can say about the two confirmed concepts: • 
instructions
is used for system-level guidance you want the agent to follow (tone, goals, constraints, persona, or extra context to shape replies). • A parameter named like 
user_input
(if present in your SDK) is typically where you pass the actual user message that the agent should respond to. Next steps — pick one of these so I can give exact guidance: • Share the AgentSession 
generate_reply
method signature or a link to the SDK docs you’re using, or • Tell me which meaning of 
user_data
you intended (raw user message vs structured context). If 
user_data
is structured context and your SDK doesn’t support it directly, I can show two practical options: • Put the context inside 
instructions
as background the agent should use, or • Serialize the context and include it alongside the user message in 
user_input
(with an explicit cue like “Context: ...”) Tell me which option matches your use case and I’ll provide a concrete example call. , with interactive elements
b
Sorry I meant 
generate_reply(user_input="...")
v.s. 
generate_reply(instructions="...")
t
Thanks — that clarification helps. I'll explain the difference between 
generate_reply(user_input="...")
and 
generate_reply(instructions="...")
in an AgentSession and when to use each. • 
user_input
 ◦ Represents an actual user message that the agent should reply to. ◦ The text is added to the chat history as a user message (so it becomes part of the conversation context). ◦ Use this when you want to simulate or forward a real user's message, or when the agent should respond directly to content that should persist in the session history. ◦ Example:
Copy code
await session.generate_reply(user_input="I need help with my account")
• 
instructions
 ◦ Provides system-level guidance for how the agent should generate the reply (tone, behavior, tasks, or one-off directives). ◦ This guidance is not recorded as a user message in chat history; only the generated agent reply is stored. ◦ Use this when you want to shape the agent's behavior (e.g., “Greet warmly and ask a clarification question”) or trigger an agent-initiated message that should not appear as a user message in history. ◦ Example:
Copy code
await session.generate_reply(instructions="Introduce yourself and offer help")
Key practical differences • History: 
user_input
is stored as a user message; 
instructions
are not stored in history. • Purpose: 
user_input
= content to be replied to; 
instructions
= internal/system prompt shaping the reply. • Typical use cases: Use 
user_input
for real user content; use 
instructions
for agent-initiated actions, system prompts, or temporary guidance. If you want, tell me which behavior you need (e.g., forward messages from another source, or have the agent proactively greet users), and I’ll give a concrete code example tailored to that use case. Sources: Agent speech and audio | LiveKit Documentation | Amazon Nova Sonic integration guide | LiveKit Documentation, with interactive elements
b
When we call 
generate_reply(user_input="...")
, it adds the input to chat history. What happens if we keep calling 
generate_reply
with large size of user_input? Does it overflow the context? Or does LiveKit handle the truncation nicely?
t
Thanks — good question. You’re asking what happens if you keep calling 
generate_reply(user_input="...")
with large inputs and whether that can overflow context or if LiveKit truncates it for you. Short answer: • Repeatedly calling 
generate_reply(user_input=...)
appends those messages to the session’s chat history. • LiveKit does not automatically truncate or protect against context/window overflow for you. Context management is the developer’s responsibility. What that means in practice: • If the accumulated history grows beyond the model’s context window, you can hit the model’s token limit (causing errors or degraded performance) or face larger latency and cost. • LiveKit provides APIs and helpers you can use to manage context (for example, copying and truncating a 
ChatContext
with 
chat_ctx.copy().truncate(max_items=...)
, or producing a summary of prior turns), but it won’t silently drop old user messages for you. Recommended approaches to avoid overflow: 1. Periodic truncation ◦ Keep only the last N turns using 
truncate()
when you detect the history growing too large. 1. Summarization ◦ Periodically summarize older conversation turns into a short context note, replace older messages with that summary, and continue. 1. Hybrid approach ◦ Keep recent turns verbatim and summarize older history to preserve important context without blowing the token budget. 1. Monitoring ◦ Track token count or message count and trigger truncation/summarization before hitting limits. If you’d like, I can: • Show a concrete code example for truncating 
chat_ctx
before calling 
generate_reply
, or • Provide a sample flow that summarizes older turns and keeps only the summary plus recent messages. Which would help you most? Sources: Agent speech and audio | LiveKit Documentation | Agents and handoffs | LiveKit Documentation, with interactive elements
b
Can we make 
generate_reply()
always call a tool?
t
Thanks — I’m responding to your question about whether you can make 
generate_reply()
always call a tool. Short answer • You cannot force 
generate_reply()
itself to always call a tool unconditionally. Whether a tool is invoked is decided by the LLM (how it’s prompted and how tools are configured). • You can, however, strongly influence or require tool usage by configuring the agent/LLM and instructions so the model must call a tool. What controls tool invocation • The LLM decides whether to call a tool based on: ◦ The system instructions you provide (via 
instructions
or the agent’s system prompt). ◦ The available tools you register with the agent. ◦ The LLM provider’s tool selection settings (for example, some providers support a 
tool_choice
option you can set to 
'required'
). • 
generate_reply()
is the API that triggers generation; it doesn’t contain a built-in flag to force a tool call independent of the LLM and tool configuration. Practical ways to ensure a tool is called • Make the tool explicitly required in system instructions: ◦ Use 
instructions
that clearly tell the agent to always use the tool for the task (e.g., “Always call the X tool to fetch live data before replying.”). • Use provider-specific tool enforcement: ◦ If your LLM provider supports a 
tool_choice
or similar setting (examples exist for some providers), set it to require tool use so the model must pick a tool if available. • Build a wrapper that enforces tool calls: ◦ Before calling 
generate_reply()
, call the tool yourself and inject the tool result into the conversation or into 
instructions
. Then call 
generate_reply()
so the agent replies using the tool output. • Use a two-step pattern: a. Run a short model step that decides which tool to call. b. Execute that tool and append the tool result to chat history or instructions. c. Call 
generate_reply()
to produce the final user-facing reply. Which approach to choose • Want the model to decide and always use a tool if available? Try 
tool_choice='required'
(if your LLM provider supports it) plus clear system 
instructions
. • Want to guarantee the tool runs every time regardless of model behavior? Call the tool from your code and pass its output to 
generate_reply()
. If you tell me: • Which LLM provider you’re using, or • Whether you prefer the model to pick the tool vs. your code explicitly running the tool, I’ll provide a concrete example (code snippets and recommended 
instructions
) for that setup. Sources: Agent speech and audio | LiveKit Documentation | OpenAI LLM models | LiveKit Documentation | Telnyx LLM plugin guide | LiveKit Documentation | OVHCloud LLM plugin guide | LiveKit Documentation | Groq LLM plugin guide | LiveKit Documentation, with interactive elements
b
Is there a way to dynamically set the available tools when calling 
generate_reply()
?
t
Thanks — you’re asking whether you can dynamically set the available tools when calling 
generate_reply()
. Short answer • You cannot pass a new tool list directly into a single 
generate_reply()
call. The toolset the model can use is controlled by the Agent’s state, not a per-call parameter on 
generate_reply()
. How to dynamically change which tools are available • Update the Agent before calling `generate_reply()`: ◦ Use 
agent.update_tools()
(or 
agent.updateTools()
in Node) to replace, add, or remove tools at runtime. ◦ After you update the agent’s tools, subsequent 
generate_reply()
calls will see the new toolset. Examples • Python (replace all tools or add/remove):
Copy code
# Replace all tools
await agent.update_tools([tool_a, tool_b])

# Add a tool
await agent.update_tools(agent.tools + [new_tool])

# Remove a tool
new_tool_list = [t for t in agent.tools if t.name != "tool_to_remove"]
await agent.update_tools(new_tool_list)
• Node.js (update tool context):
Copy code
// Add or replace keys in toolCtx then call updateTools
await agent.updateTools({ ...agent.toolCtx, newTool })
Alternative / complementary options • Use 
instructions
to guide tool usage: You can instruct the agent to call a particular tool (for example, “Use the check_order_status tool”), but that requires the tool to already be present on the Agent. • Enforce tool usage from your code: If you need guaranteed tool execution every time, call the tool from your application first and inject the tool output into the session (either by appending it to chat history or placing it into 
instructions
) before calling 
generate_reply()
. When to choose which approach • You want the model to choose from a different toolset per request: update the Agent’s tools before each 
generate_reply()
call. • You need the tool to definitely run (not left to model decision): run the tool in your code and pass results into the session. • You want to bias or require tool usage without changing tools: use explicit 
instructions
and, if supported by your LLM provider, set provider-specific options like 
tool_choice='required'
. If you tell me which runtime you’re using (Python or Node) and whether you want tools swapped per-user, per-room, or per-request, I can provide a concrete code pattern you can plug into your app. Sources: Function tool definition | LiveKit Documentation | Agent speech and audio | LiveKit Documentation, with interactive elements
b
Can we modify a tool raw_schema before calling 
generate_reply()
?
t
You can modify a tool's 
raw_schema
before calling 
generate_reply()
. What to know • Tools defined from a raw function-calling schema accept a 
raw_schema
(a dict/JSON) when created, so you can programmatically change that schema at runtime before the tool is registered with an agent. • Because 
raw_schema
is a plain data structure, you can add/remove fields, change 
description
, alter 
parameters
, set 
enum
values, etc., to adapt the tool to runtime conditions (user preferences, locale, permissions, etc.). • Changes must be applied before the Agent uses the tool. If the tool is already registered on an active Agent, update the agent's tools after changing the schema so the Agent sees the update. How to apply it (patterns) • Create or modify the 
raw_schema
dict, then create the tool with it. • Register the tool with the Agent (or update the Agent’s tools) before calling 
generate_reply()
. • If the Agent is already using the tool, call 
agent.update_tools()
(or 
agent.updateTools()
in Node) with the new/updated tool so subsequent 
generate_reply()
calls use the modified schema. Python example
Copy code
from livekit.agents import function_tool

# start with base schema
schema = {
    "type": "function",
    "name": "get_weather",
    "description": "Get weather for a location.",
    "parameters": {
        "type": "object",
        "properties": {
            "location": {"type": "string", "description": "City and country"}
        },
        "required": ["location"]
    }
}

# modify schema dynamically
schema["description"] = "Get detailed weather forecast for a city."
schema["parameters"]["properties"]["unit"] = {
    "type": "string",
    "enum": ["celsius", "fahrenheit"]
}

# create tool with modified schema
tool = function_tool(handler, raw_schema=schema)

# register/update agent tools before calling generate_reply()
await agent.update_tools(agent.tools + [tool])

# now generate_reply() will see the tool with the modified schema
await session.generate_reply(user_input="What's the weather in Paris?")
Notes and best practices • Ensure the schema matches the expected function-calling format (OpenAI-style) and that your tool handler can process any added/changed parameters. • If you change schemas frequently, manage versions or clear naming collisions so the Agent uses the intended tool. • If you need guaranteed behavior for every request (rather than relying on the LLM to call the tool), consider invoking the tool from your code and injecting results into the session before 
generate_reply()
. If you want, tell me: • Whether you’re using Python or Node, and • If you need a sample for updating a tool in an already-running Agent (I’ll provide a concrete snippet showing 
update_tools()
and a safe update pattern). Sources: Function tool definition | LiveKit Documentation, with interactive elements
Open in Slack
PreviousNext
Powered by