https://livekit.io logo
Join Slack
Powered by
how are my agent's tools passed to llm to get a re...
# ask-ai
c
how are my agent's tools passed to llm to get a response
✅ 1
t
Thanks for reaching out, Kushagra. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — you're asking how your agent's tools get passed to the LLM so it can produce a response. Here's a concise explanation of that flow and what you control. How tools are passed to the LLM • Tools are defined on the agent (Python 
@function_tool
or Node 
llm.tool()
/ Zod schemas). Each tool includes a name, description, and typed input schema. • When the agent initializes a call to the LLM, the agent supplies a structured list of those tool schemas via the 
tools
parameter. The LLM sees the tool names, descriptions, and parameter schemas so it can decide to invoke one. • At runtime: user audio → STT → transcribed text sent to the LLM (with the tool schemas). The LLM can return either a direct text reply or a tool call (structured output). • If the LLM issues a tool call, the framework intercepts it, extracts the parameters, executes the corresponding tool implementation, returns the tool result to the LLM, and the LLM then generates the final natural-language response using that result. Additional notes • Function-tool schemas are generated from the function signature/docstrings (Python) or Zod/JSON schemas (Node). • Provider-side tools (e.g., WebSearch) can be added the same way and are exposed to the LLM as plugins in the tools list. • Tools can be added/removed dynamically (update_tools) so the set of available tools the LLM sees can change per conversation. If you want, tell me: • which language (Python or Node) you’re using, and • an example tool you’ve defined and I’ll show the exact decorator/schema and the minimal payload that gets sent to the LLM. Sources: Function tool definition | LiveKit Documentation | Tool definition and use | LiveKit Documentation | Sequential Pipeline Architecture for Voice Agents | LiveKit, with interactive elements
c
what is the structure when the tools are passed as payload to llm for response generation
t
Thanks — you're asking for the exact payload structure used when LiveKit passes agent tools to the LLM. Below is the concise structure and a clear example. Tool payload structure (OpenAI-style function schema) • The request includes a 
tools
array; each tool is an object with: ◦ `type`: "function" ◦ `function`: { ▪︎ `name`: string, ▪︎ `description`: string, ▪︎ `parameters`: { • `type`: "object", • `properties`: { param_name: { "type": type, "description": string }, ... }, • `required`: [ ... ], • `additionalProperties`: false } } Example JSON entry for a 
lookup_weather
tool
Copy code
{
  "type": "function",
  "function": {
    "name": "lookup_weather",
    "description": "Look up weather information for a given location.",
    "parameters": {
      "type": "object",
      "properties": {
        "location": {
          "type": "string",
          "description": "The location to look up weather information for."
        },
        "units": {
          "type": "string",
          "description": "Units for temperature (metric or imperial)."
        }
      },
      "required": [
        "location"
      ],
      "additionalProperties": false
    }
  }
}
Runtime flow summary 1. Agent sends user text (after STT) to the LLM along with the 
tools
array above. 2. The LLM may return a tool call (name + JSON args) instead of plain text. 3. The framework executes the matching tool with those args and returns the result to the LLM. 4. The LLM produces the final natural-language response using the tool output. If you want, tell me which language (Python or Node) you're using and one example tool signature — I’ll show the exact decorator/schema and the minimal request payload sent to the LLM. Sources: OpenAI LLM models | LiveKit Documentation | Tool definition and use | LiveKit Documentation | Function tool definition | LiveKit Documentation, with interactive elements
c
what if i write the tool as
Copy code
def tool_name(self, params):
    """ some description """
    ...tool body
t
Thanks — you're asking whether defining a tool as 
def tool_name(self, params): ...
will work and what the implications are. Short answer and guidance: • That signature will run as regular code, but LiveKit will not automatically generate a structured schema for the LLM from a single untyped 
params
argument. The LLM needs a JSON-schema/function schema to know what arguments to provide. • To expose the tool to the LLM and get function-calling behavior, declare it with the function-tool mechanism so the schema is generated or provided: ◦ In Python: use 
@function_tool
on a method or function and use typed parameters or provide a 
raw_schema
. Example signatures generate proper schemas from type hints and docstrings. ◦ In Node: use 
llm.tool()
with a Zod or JSON schema for 
parameters
. If you must accept a single 
params
dict, provide an explicit schema (via 
raw_schema
in Python or a JSON/Zod schema in Node) that describes the expected keys and types — otherwise the LLM won't know the structure and won't produce reliable tool calls. Tell me which language (Python or Node) and I’ll show a minimal example converting your 
params
style into a properly exposed tool schema. , with interactive elements
c
create an example scenario and explain
t
I'll show a concrete scenario in Python that matches your 
def tool_name(self, params)
style, then explain how to expose a JSON schema so the LLM can call it, and show the runtime payload and flow. Scenario • Goal: Agent provides user account details when asked (tool accepts a single 
params
dict with 
user_id
and 
fields
). • Problem: A plain 
params
argument has no type info for the LLM. You must supply a schema so the LLM can generate structured tool calls. 1) Tool implementation (Python, method with 
params
)
Copy code
from livekit.agents import function_tool

class AccountAgent:
    @function_tool(raw_schema={
        "name": "get_account_info",
        "description": "Return account fields for a given user id.",
        "parameters": {
            "type": "object",
            "properties": {
                "user_id": {"type": "string", "description": "ID of the user"},
                "fields": {
                    "type": "array",
                    "items": {"type": "string"},
                    "description": "List of account fields to return, e.g. ['email','plan']"
                }
            },
            "required": ["user_id"],
            "additionalProperties": False
        }
    })
    async def get_account_info(self, params):
        # params is a dict {"user_id": "...", "fields": ["..."]}
        user_id = params.get("user_id")
        fields = params.get("fields", ["email"])
        # ...fetch data from DB or service...
        return {f: f"dummy-{f}-for-{user_id}" for f in fields}
Notes: • 
@function_tool(raw_schema=...)
explicitly supplies the function schema the LLM needs. • The implementation still receives a single 
params
dict, so your existing logic can stay the same. 2) Example tool entry LiveKit sends to the LLM (JSON function schema)
Copy code
{
  "type": "function",
  "function": {
    "name": "get_account_info",
    "description": "Return account fields for a given user id.",
    "parameters": {
      "type": "object",
      "properties": {
        "user_id": {
          "type": "string",
          "description": "ID of the user"
        },
        "fields": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "description": "List of account fields to return"
        }
      },
      "required": [
        "user_id"
      ],
      "additionalProperties": false
    }
  }
}
3) Runtime flow (end-to-end) 1. User: "What is the email and plan for user 123?" 2. Audio → STT → transcribed text sent to LLM along with the 
tools
array containing the schema above. 3. LLM decides to call 
get_account_info
and returns a function call like: ◦ name: "get_account_info" ◦ arguments: {"user_id":"123","fields":["email","plan"]} 1. LiveKit framework intercepts the tool call, invokes 
AccountAgent.get_account_info(params)
with 
params
= that dict. 2. Tool returns the data (e.g., {"email":"dummy-email-for-123","plan":"dummy-plan-for-123"}). 3. Framework returns the tool result to the LLM; the LLM generates the final natural-language reply, e.g., "User 123 has email ... and is on plan ...", which is sent back to the user (TTS if voice). If you prefer Node or want an example where the schema is derived from typed parameters (no 
params
dict), tell me which and I’ll show that minimal example. , with interactive elements
c
what if i write it like this
Copy code
@function_tool()
async def tool_name(self, context: Context):
    """ use this tool ... """
    <tool body>
t
Thanks — you're asking whether this signature works and how it's exposed to the LLM:
Copy code
@function_tool()
async def tool_name(self, context: RunContext):
    """ use this tool ... """
    <tool body>
Short answer • Yes — this is valid. The framework accepts a 
context
(use 
RunContext
) and the tool runs normally. • The LLM will see this tool as accepting no user arguments. LiveKit generates a function schema with empty 
properties
and no 
required
fields, so the LLM can call it without arguments. What 
RunContext
is for • 
RunContext
is passed to your tool at runtime (not part of the LLM-visible schema). It provides access to session state, 
userdata
, methods like 
disallow_interruptions()
and 
wait_for_playout()
, and other runtime helpers the tool can use internally. Example generated schema sent to the LLM
Copy code
{
  "type": "function",
  "function": {
    "name": "tool_name",
    "description": "use this tool ...",
    "parameters": {
      "type": "object",
      "properties": {},
      "required": []
    }
  }
}
Runtime flow (brief) 1. User input → STT → text + tools array (including schema above) sent to LLM. 2. LLM may return a function call: name = "tool_name", with empty/absent arguments. 3. Framework invokes 
tool_name(self, context)
passing a 
RunContext
. 4. Tool returns a value (string or structured data); result is passed back to the LLM for final reply. When to use this pattern • Use it when the tool performs a fixed action or uses/contributes to session state and doesn’t need user-supplied parameters. • If you need user-provided inputs, add typed parameters (or supply 
raw_schema
) so the LLM knows what arguments to provide. If you want, tell me whether you want: • an example that returns structured data, or • the variant that accepts user args (typed params or 
raw_schema
) and I’ll show the minimal code + request payload. Sources: Function tool definition | LiveKit Documentation, with interactive elements
6 Views
Open in Slack
PreviousNext
Powered by