https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# ask-ai
s
This message was deleted.
t
Thanks for reaching out, Aleh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great question and your code snippet made the issue clear. Quick answer • When a single turn calls two tools, the agent only receives each tool’s explicit return values as tool outputs. Your first tool returns the string "Application confirmed." which becomes a tool output. Your second tool calls 
session.generate_reply()
which performs audio/speech generation but does not return a value to the LLM (it returns None). Because 
generate_reply()
doesn’t produce a string result back into the tool outputs, the agent has nothing from that tool to include in the final text reply. Why this happens • 
session.generate_reply()
is intended to produce speech playback or trigger an agent-side reply during turn handling or lifecycle hooks; it’s not a mechanism for returning structured text back to the LLM inside a tool call. The framework collects explicit return values from tools to build the final response; side effects like audio playback are not captured as tool outputs. What you can do (two straightforward options) 1) Return the KB answer as a string from the tool (recommended) • Change 
access_knowledge_base
to return the generated answer text instead of calling 
session.generate_reply()
inside the tool. • Let the framework pass both returned strings (from 
confirm_application_details
and 
access_knowledge_base
) to the LLM so it can generate a single cohesive response. Example:
Copy code
@function_tool()
async def access_knowledge_base(self, context: RunContext[SessionState], question: str) -> str:
    # ... build knowledge_instruction or the answer text ...
    return knowledge_answer_text  # return the KB answer as a string
2) If you need both text for the LLM and audio playback • Have 
access_knowledge_base
return the answer text, and then trigger 
session.generate_reply()
from a place that the framework expects side-effecting speech (for example, after the tool returns, in the agent response handler or a lifecycle hook). This ensures the LLM receives the text and the user still hears audio. Next step I can help with • If you’d like, paste the code you use to build 
knowledge_instruction
or the exact text you want spoken and I’ll show the exact code change to return the KB answer and optionally trigger speech after the tool returns. Sources: Testing and evaluation | LiveKit Documentation | Tool definition and use | LiveKit Documentation | Agent speech and audio | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation
b
i can't do it this way, because here i get user question and llm check what it have in KB and generate answer there is no sense to return all KB content to the llm to decide what to answer
t
Thanks — I hear your constraint: you need the LLM to consult the KB internally and decide the answer, and you don’t want to dump the entire KB content back into the LLM as a returned string. A few important facts to keep in mind • 
session.generate_reply()
produces audio/speech as a side effect but does not return text back to the LLM. Inside a tool, its side effect is executed but the LLM still only receives the tool’s explicit return value (or None) as the tool output. • The framework builds the final agent reply from the explicit tool return values. So if your KB tool does not return a text value, the LLM has no KB-derived text to include in the final reply. Possible patterns that preserve LLM decision-making without returning full KB content • Return a concise, structured summary/result instead of the whole KB ◦ Have your KB tool run the internal retrieval + reasoning and return just the final answer text (or a short structured object like {"answer": "...", "source_ids": [...]}) rather than returning full KB context. ◦ This keeps the KB lookup and reasoning inside the tool but still provides the LLM with a text result to incorporate in the final reply. • Return an instruction-style answer for the LLM to adopt ◦ Let the KB tool produce the canonical response text (the text you want the LLM to speak/use). Return that short canonical text. The LLM can then weave it into the turn or rephrase if needed. ◦ This prevents exposing raw KB documents but still gives the LLM the knowledge it needs. • Use a two-step flow: tool returns a decision token + ephemeral session context ◦ Tool returns a small token/flag + metadata (e.g., 
{"kb_decision":"approved","summary":"X is the answer","confidence":0.92}
). ◦ After tools finish, an agent-side lifecycle hook or the agent’s turn handler can assemble the spoken response using that summary and call 
session.generate_reply()
outside the tool so speech and final text align. This separates speech side-effects from the KB lookup while keeping the LLM aware of the KB-derived decision. • Keep reasoning inside the tool but return the answer only when the LLM needs to craft tone/format ◦ If you want the LLM to control tone/phrasing, return a concise canonical answer from the tool and let the LLM rephrase. If you want deterministic wording (for e.g., compliance), return the exact text and let the LLM just pass it through. Questions to pick the best pattern for you 1. Do you need the LLM to rephrase the KB answer (flexible tone), or must the KB answer be returned verbatim (regulatory/compliance)? 2. Do you also need speech playback in the same turn? If yes, should the speech match the LLM’s final phrasing or the KB tool’s canonical wording? 3. Is metadata useful (e.g., sources, confidence) to be returned along with the concise answer so the LLM can decide whether to disclose sources? If you tell me: • whether you need rephrasing vs. verbatim text, • and whether speech must be triggered inside the tool or can be triggered after the tool returns, I’ll provide one concrete code pattern that implements it (example for returning a concise answer + triggering speech outside the tool, or example for returning a structured object and using a lifecycle hook to play audio).
2 Views
Open in Slack
PreviousNext
Powered by