https://livekit.io logo
Join Slack
Powered by
How can I tell the agent within a tool to find a v...
# ask-ai
a
How can I tell the agent within a tool to find a value in a json object? I don't want the agent to generate any audio for this, but get the response so I can act on it.
✅ 1
t
Thanks for reaching out, Ahron. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I understand you want the agent to extract a value from a JSON object inside a tool and return it programmatically without generating any audio. Here’s how to do that. 1. In your tool implementation, parse the JSON and extract the value you need in code. 2. Do NOT call any speech APIs such as 
session.say()
or 
session.generate_reply()
inside the tool — leaving those out prevents audio output. 3. Return the extracted value (or a structured object) from the tool so the agent/LLM receives it for further programmatic handling. ◦ If you want the tool to act silently and not prompt any LLM reply, you can return 
None
after storing/using the value internally. 1. If the JSON lives on the client, use a client RPC tool to fetch it, apply steps 1–3 in that tool, and return the value silently. If you’d like, tell me: • where the JSON is (tool backend vs frontend), • an example JSON and the key you want, and I’ll give a short code example tailored to your setup. Sources: Function tool definition | LiveKit Documentation | Agent Builder | LiveKit Documentation, with interactive elements
a
I need an ai to find the matched value since the input isn't the exact value.
t
Thanks — I can help. A couple quick clarifying questions so I give the right approach: • Do you want fuzzy matching (typo-tolerant), partial substring matching, or semantic / embedding similarity? • Where will this run: in a backend tool function or a client RPC tool? • Can you share a small example of the JSON and an example input string (anonymized is fine)? With those I’ll provide a concise code example and recommended matching method. , with interactive elements
a
no I need to provide the ai the input to give me the output
t
Thanks — I can help. Quick clarification so I give exactly what you need: • Which matching method do you want the AI to use: fuzzy/typo-tolerant (e.g., Levenshtein), partial substring, or semantic/embedding similarity? • Where will this run: a backend tool function or a client-side RPC tool? • Please share a small example JSON and one example input string (anonymized is fine). Once you provide those, I’ll give a concise, silent tool code example that returns the matched value programmatically. , with interactive elements
a
forget about the details. All I'm asking how can I make a request to the agent within a tool without generating audio
t
Understood — here’s a short, direct answer. 1) In your tool code, extract/process the JSON and return the result programmatically. 2) Do NOT call any speech methods (
session.say()
or 
session.generate_reply()
) in the tool — omitting those prevents audio. 3) If you want the tool to be entirely silent to the LLM while still acting, you can store/use the value internally and return `None`; otherwise return the extracted value so the agent receives it for further logic. Minimal example (backend tool):
Copy code
def find_value_tool(input_json, query):
    # extract/match value (example: exact or fuzzy logic here)
    matched = my_matching_function(input_json, query)
    # do NOT call session.say() or session.generate_reply()
    return matched  # or return None if you want no LLM response at all
If JSON is client-side, call a client RPC tool that performs the same steps and returns the matched value without invoking speech. If you want, share one line of your tool environment (backend vs client) and I’ll give a 1–2 line code example that fits it. , with interactive elements
a
Use AgentTask
t
Got it — here’s a concise example showing how to use AgentTask to ask the agent to find a value in a JSON object without generating audio. Key point: do not call any speech methods (
session.say()
/ 
session.generate_reply()
). Python (tool) example:
Copy code
from livekit_tools import AgentTask  # adjust import to your SDK

def agent_find_json_value(agent_client, json_obj, query):
    task = AgentTask(
        instruction=f"Find the best-matching value in this JSON for: {query}",
        input={"json": json_obj, "query": query},
        # ensure task settings don't auto-generate speech (if SDK has such flags)
    )
    # run the agent task and get its programmatic response
    result = agent_client.run_task(task)
    # result contains the agent's returned value (no audio generated if tool avoids speech calls)
    return result  # could be a string, object, or None
Notes: • Inside the agent/tool logic, parse the 
input
and return the matched value without calling 
session.say()
or 
session.generate_reply()
. • If you want the agent completely silent to the LLM loop, have the tool store/use the value and return `None`; otherwise return the value so you can act on it programmatically. If you want, tell me whether you prefer a short example using exact/fuzzy/semantic matching and I’ll provide the matching snippet. , with interactive elements
a
Can I manually call an llm?
t
Yes — you can manually call an LLM inside a tool (or via AgentTask) and keep it silent. The key is: perform the LLM call programmatically and do not invoke any speech methods (
session.say()
/ 
session.generate_reply()
). Quick examples: 1) Using AgentTask (already shown) — run the task and read the returned value; ensure the agent/tool implementation does not call speech. 2) Direct LLM call (pseudo-Python):
Copy code
# call the LLM client directly from your tool
response = llm_client.generate(
    messages=[{"role":"system","content":"Extract the best matching value."},
              {"role":"user","content":f"JSON: {json_obj}\nQuery: {query}"}],
    # any other params...
)
# do NOT call session.say() or session.generate_reply()
value = parse_response(response)  # extract the matched value programmatically
return value  # or return None if you want no LLM/agent reply at all
If you tell me which SDK/language you’re using (Python, Node, or specific LiveKit client), I’ll give a 1–2 line exact snippet for that environment. , with interactive elements
a
how do I get the llm_client?
t
You can get an LLM client two ways and call it silently inside a tool or AgentTask — then avoid any speech calls. 1) Inside a tool/agent (recommended): use the session LLM
Copy code
# inside a @function_tool or AgentTask method with RunContext or self.session
llm_client = ctx.session.llm   # or self.session.llm
chat_ctx = llm.ChatContext()
chat_ctx.add_message(role="user", content=f"JSON: {json_obj}\nQuery: {query}")
response = await llm_client.chat(chat_ctx=chat_ctx).collect()
value = parse_response(response.text)
return value  # no session.say()/session.generate_reply() => no audio
2) Instantiate directly (standalone)
Copy code
from livekit.agents import inference, llm
llm_client = inference.LLM(model="openai/gpt-4.1-mini")
response = await llm_client.chat(chat_ctx).collect()
Key points: • Use 
ctx.session.llm
or 
self.session.llm
when inside agent/tool code. • Do not call 
session.say()
or 
session.generate_reply()
— omitting those prevents audio. • Return the extracted value (or return 
None
if you want completely silent completion). Sources: Bring your LangChain Agents to LiveKit | LiveKit | Large language models (LLM) overview | LiveKit Documentation, with interactive elements
a
I don't want this to be part of the current session chat
t
Understood — here’s how to call the LLM outside the current session chat and keep it silent. 1) Instantiate a standalone LLM client (no session context):
Copy code
from livekit.agents import inference, llm

llm_client = inference.LLM(model="openai/gpt-4.1-mini")
chat_ctx = llm.ChatContext()
chat_ctx.add_message(role="user", content=f"JSON: {json_obj}\nQuery: {query}")

response = await llm_client.chat(chat_ctx=chat_ctx).collect()
value = parse_response(response.text)
# no session.say() or session.generate_reply() — nothing is spoken
2) Why this stays out of session chat: • You created and used a separate 
llm_client
instance instead of 
ctx.session.llm
or 
self.session.llm
, so the call is not recorded in the active session chat state. 3) Checklist to ensure silence and separation: • Use a standalone LLM instance (as above). • Do not call 
session.say()
or 
session.generate_reply()
. • Return or store the parsed value programmatically (or return 
None
if you want no agent reply). If you want a 1–2 line snippet for Node or the exact matching prompt to extract values, tell me which language and I’ll provide it. , with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by