This message was deleted.

Thanks for reaching out, Aleh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

@function_tool()
async def confirm_application_details(self, context: RunContext[SessionState]) -> str:
    """Mark application details as confirmed by customer."""
    self.application_confirmed = True
    context.userdata.call_outcome.application_confirmed = True
    return "Application confirmed."

@function_tool()
async def access_knowledge_base(self, context: RunContext[SessionState], question: str) -> None:
    """Answer customer questions about company services, loans, or financial products.

    Use this tool when the customer asks questions about:
    - Loan products, rates, terms, or conditions
    - Total balance, total payback amount, or total payment
    - Monthly payment amount
    - Approved loan amount or pre-approved amount
    - Company services and offerings
    - Account information or loan details
    - Branch locations or contact information
    - Any finance or lending related topics

    Args:
        question: The customer's question about company/loan topics
    """
    userdata = context.userdata
    company_name = userdata.company_name
    customer_full_name = f"{userdata.customer_data.firstname} {userdata.customer_data.lastname}"
    is_verified = userdata.call_outcome.identity_verified

    general_kb_vars = {
        "servicing_branch_name": userdata.servicing_info.servicing_branch_name,
        "servicing_branch_phone_number": userdata.servicing_info.servicing_branch_phone_number,
    }
    general_kb_content = general_knowledge_base_content.invoke(general_kb_vars).text
    knowledge_content_text = general_kb_content
    if is_verified:
        sensitive_kb_vars = {
            "pre_approved_loan_amount": userdata.loan_info.pre_approved_loan_amount,
            "loan_apr": userdata.loan_info.loan_apr,
            "monthly_payment": userdata.loan_info.monthly_payment,
            "total_payback_amount": userdata.loan_info.total_payback_amount,
            "loan_term_months": userdata.loan_info.terms,
        }
        sensitive_kb_content = sensitive_knowledge_base_content.invoke(sensitive_kb_vars).text
        knowledge_content_text = f"{general_kb_content}\n\n{sensitive_kb_content}"

    knowledge_instruction = base_agent_access_knowledge_base_instructions.invoke(
        {
            "company_name": company_name,
            "customer_full_name": customer_full_name,
            "question": question,
            "knowledge_base_content": knowledge_content_text,
            "servicing_branch_phone_number": general_kb_vars["servicing_branch_phone_number"],
        }
    ).text

    await self.session.generate_reply(instructions=knowledge_instruction)

Thanks — this is clear and helpful. You showed the two tools and the exact failure mode (a single user utterance that both confirms and asks a question causes only one tool to run). I’ll focus on practical, low-friction changes you can make now and give a concrete example you can drop into your code. Quick recap of what I’m addressing • You have a

confirm_application_details

tool that sets state when a customer confirms. • You have an

access_knowledge_base

tool that answers finance/product questions. • When the user says something like “Yes, that’s correct. What’s my monthly payment?” the confirmation often isn’t run and your workflow breaks. What you can do next (ordered from simplest to most robust) 1) Short-term: Make confirmation idempotent and persist state • Ensure confirming the application sets state in

userdata

(you already do this) and that downstream logic checks

userdata.call_outcome.application_confirmed

rather than relying only on the tool having been called in the same turn. • This makes the system tolerant if the question tool runs first — the confirmation state can still be set when later detected. 2) Low-effort: Ask for an explicit follow-up when ambiguity is detected • Change

confirm_application_details

reply to both confirm and invite the question: “Application confirmed. Would you like me to answer a question about payments now?” • This pushes the user into a second turn where

access_knowledge_base

will be selected reliably. 3) Deterministic orchestration (recommended): Add an intent-extraction step and orchestrate tool calls yourself • Add a small pre-processing step that detects intents in the incoming user text (confirmation intent, question intent). • Based on extracted intents, your orchestration code explicitly calls the tools in the order you choose: a. Call

confirm_application_details

if confirmation intent present. b. Call

access_knowledge_base

if question intent present (pass the extracted question text). • This removes reliance on the LLM choosing the right tool when inputs are compound. 4) Composite tool (if you prefer a single tool call) • Implement a new tool that handles the combination in one place:

@function_tool()
async def confirm_and_answer(self, context: RunContext[SessionState], question: Optional[str] = None) -> str:
    """Confirm application details and optionally answer a question included in the same user message."""
    # Confirm
    self.application_confirmed = True
    context.userdata.call_outcome.application_confirmed = True

    # If a question is provided, answer it via the knowledge base tool
    if question:
        await self.access_knowledge_base(context, question)
        return "Application confirmed and question answered."
    return "Application confirmed."

• Update agent instructions so the model prefers this tool when a message contains both confirm + question. 5) Parallel tool calls (only if your LLM backend supports it) • If your provider supports parallel tool calls, enable that capability so the LLM can invoke both tools in a single turn. If that’s available, you still want clear orchestration rules or a composite tool to guarantee ordering (confirm first, then answer). Concrete adjustments to your current code and prompts • Add explicit intent hints in tool docstrings so the LLM can detect both intents: ◦ In

confirm_application_details

docstring: “If the user confirms and also asks a question in the same message, either trigger the composite tool

confirm_and_answer

or confirm first and then call the knowledge tool.” ◦ In agent-level system instructions: “When a single user message includes both confirmation and a question, ensure the confirmation is recorded first. Then answer the question.” • Implement a simple intent parser (regex / small classifier) prior to calling the agent tools: ◦ Detect confirmation phrases: “yes”, “correct”, “that’s right”, “confirmed”, etc. ◦ Detect question marks or interrogative phrases and extract the question string. ◦ Use these signals to call tools deterministically. Which approach should you pick? • If you want the fastest / lowest-risk change: make confirmation idempotent, persist state, and invite a short clarifying follow-up. This avoids major refactors. • If you want reliability and deterministic behavior: add the intent-extraction + orchestration or create the composite

confirm_and_answer

tool. • If your LLM supports parallel calls and you prefer that UX, enable it but keep deterministic checks to guarantee ordering. If you want, I can: • Provide a ready-to-drop intent extraction snippet (Python) that extracts confirmation and question text from a user input. • Show a full example implementing the composite tool and the small orchestration wrapper that calls the current tools in sequence. Which follow-up would be most helpful to you now — the intent-extraction snippet, the composite-tool full example, or guidance for enabling parallel tool calls with your LLM provider? Sources: Tool definition and use | LiveKit Documentation | Building multi-agent architectures with LiveKit agents | LiveKit | DeepSeek LLM | LiveKit Documentation | Tasks and task groups | LiveKit Documentation | Azure OpenAI LLM plugin guide | LiveKit Documentation

Option 1 (Idempotent state): Doesn't fix anything.

applicationConfirmed

is still

False

— the tool was never called. Option 2 (Invite follow-up): Degrades UX with an extra roundtrip. The whole problem is in a single utterance. Option 3 (Intent extraction + orchestration): Architecturally the best, but requires on_user_turn_completed — confirmed unavailable in realtime setup. Dead end. Option 4 (Composite

confirm_and_answer

tool): Works but creates a second tool for confirmation, so now there are two confirmation paths (original + composite). The LLM has to choose between them and the same ambiguity shifts to a new place. Option 5 (Parallel tool calls): Azure OpenAI Realtime does support parallel function calls, but ordering is not guaranteed — access_knowledge_base could still run before confirm_application_details.

Thanks — this is really helpful context. I hear your constraints: realtime environment with no on_user_turn_completed hook, idempotent state doesn’t help because the confirmation tool never runs, follow-ups degrade UX, composite tools create duplicate confirmation paths, and parallel calls don’t guarantee ordering. Given that, let’s focus on a low-friction approach that preserves UX and guarantees the confirmation action runs first in the same single utterance. Recommended approach (single-call, deterministic ordering via structured planning) • Keep only your existing atomic tools (

confirm_application_details

and

access_knowledge_base

) and add one small, explicit orchestration mechanism that runs inside a single tool invocation the model WILL call. The trick is to create one canonical "router" tool the model can reliably choose for compound utterances and, inside that tool's implementation, deterministically call the atomic tools in the order you require (confirm first, then answer). This avoids duplicate confirmation paths at the model decision level and guarantees ordering because orchestration is done in your code, not left to the model or to parallel execution. Why this fits your constraints • Only one tool needs to be selected by the model for compound input, so you avoid the LLM picking between multiple confirmation paths. • Orchestration happens server-side within that tool, so ordering is guaranteed even if parallel function calls or turn-completion hooks are unreliable. • Single-turn UX preserved: the model returns one combined response in that same turn (or streams sub-responses from your tool if your realtime infra supports it). • No requirement for on_user_turn_completed, no changes to existing atomic tools' semantics. Concrete pattern to implement 1. Add a single router tool with a neutral name (e.g.,

handle_user_message

) and a clear docstring that instructs the LLM to call it whenever the user message might contain multiple intents (confirmation + question, updates + question, etc.). Make this the preferred tool for ambiguous/compound inputs. 2. The router tool receives the raw user_text. Inside it: ◦ Run a small deterministic intent extractor (simple rules or lightweight classifier) to identify: ▪︎ confirmation intent (yes/confirm/that’s correct/...), ▪︎ question intent (presence of interrogative words or extracted question clause). ◦ If confirmation intent present: call

confirm_application_details

programmatically (server-side). ◦ If question intent present: call

access_knowledge_base

programmatically (server-side) after confirmation call completes. ◦ Aggregate outputs from the atomic tools into a single reply and return/send that reply back into the session. 1. Keep atomic tools unchanged except ensure they are safe to be invoked programmatically from the router. Minimal example (conceptual) • Tool the model will call: ◦

handle_user_message(user_text: str) -> str

• Internal flow: a. detect confirm and question pieces from

user_text

. b. if confirm: await confirm_application_details(...) c. if question: await access_knowledge_base(..., question=extracted_question) d. compose final response: confirmation acknowledgment + knowledge answer (or just the KB answer prefixed by confirmation text) Prompting guidance for reliability • Make the router tool the single, strongly-preferred tool in your agent instructions for any message that could contain actions or compound intents. Example instruction text: “If the user input contains any action (e.g., confirm or update) or a question, call

handle_user_message

so the system can deterministically orchestrate actions and replies.” • Keep the router tool’s description concise and explicit that it should be used for compound messages to avoid the model choosing an atomic tool directly. Edge cases and behavior choices • If user_text contains only a question or only a confirmation, the router should still call the appropriate atomic tool and return the single expected reply. • If the question requires identity verification and verification fails, router should still confirm first (if present) and then return the KB response that follows your verification gating logic (your KB tool already checks verified state). • If you want streaming sub-responses (first confirmation, then KB answer streamed), implement the router so it sends confirmation text immediately and then streams the KB tool result if realtime channel supports multi-part responses. Why this is different from a composite tool • The router acts as a single decision point the model calls, but it doesn’t duplicate business logic — it programmatically invokes your canonical atomic tools. That prevents the model from choosing between multiple confirmation paths and guarantees ordering deterministically in your runtime. If you want, I can: • Provide a ready-to-drop router function implementation (Python/async) that uses simple heuristics to extract a confirmation intent and question text, and that calls your two atomic tools in order and composes the combined reply. • Help craft the exact router tool docstring and the agent system instruction lines so the LLM reliably picks the router for ambiguous utterances. Which would you prefer next — the code snippet for the router tool, or the precise instruction text to insert into your agent prompt?

164406.139 INFO livekit.agents RealtimeModel metrics {"model_name": "gpt-realtime", "model_provider": "lms-ai-sweden-dev.services.ai.azure.com", "ttft": -1.0, "input_tokens": 3261, "cached_input_tokens": 3136, "input_text_tokens": 2869, "input_cached_text_tokens": 2816, "input_image_tokens": 0, "input_cached_image_tokens": 0, "input_audio_tokens": 392, "input_cached_audio_tokens": 320, "output_tokens": 12, "output_text_tokens": 12, "output_audio_tokens": 0, "output_image_tokens": 0, "total_tokens": 3273, "tokens_per_second": 26.4} 164406.150 DEBUG livekit.agents executing tool {"function": "confirm_application_details", "arguments": "{}", "speech_id": "speech_7919f908da86"} 164406.157 DEBUG livekit.agents tools execution completed {"speech_id": "speech_7919f908da86"} 164406.816 DEBUG livekit.agents executing tool {"function": "access_knowledge_base", "arguments": "{ \n \"question\": \"What is my monthly payment?\" \n} \n", "speech_id": "speech_7919f908da86"} 164406.828 INFO livekit.agents RealtimeModel metrics {"model_name": "gpt-realtime", "model_provider": "lms-ai-sweden-dev.services.ai.azure.com", "ttft": -1.0, "input_tokens": 3284, "cached_input_tokens": 3264, "input_text_tokens": 2892, "input_cached_text_tokens": 2880, "input_image_tokens": 0, "input_cached_image_tokens": 0, "input_audio_tokens": 392, "input_cached_audio_tokens": 384, "output_tokens": 28, "output_text_tokens": 28, "output_audio_tokens": 0, "output_image_tokens": 0, "total_tokens": 3312, "tokens_per_second": 58.5} 164407.738 INFO livekit.agents RealtimeModel metrics {"model_name": "gpt-realtime", "model_provider": "lms-ai-sweden-dev.services.ai.azure.com", "ttft": 0.45, "input_tokens": 2249, "cached_input_tokens": 0, "input_text_tokens": 1857, "input_cached_text_tokens": 0, "input_image_tokens": 0, "input_cached_image_tokens": 0, "input_audio_tokens": 392, "input_cached_audio_tokens": 0, "output_tokens": 42, "output_text_tokens": 14, "output_audio_tokens": 28, "output_image_tokens": 0, "total_tokens": 2291, "tokens_per_second": 53.7} 164408.888 DEBUG livekit.agents tools execution completed {"speech_id": "speech_7919f908da86"} here i dont get speech from KB tool, why?

Thanks — this is a clear log trace, and I can see why you’re puzzled: both tools run successfully in the same turn, but you don’t get speech from the KB tool. I’ll explain the likely cause and give concrete checks/fixes you can apply right away. What I see in your logs •

confirm_application_details

was executed first and completed. •

access_knowledge_base

was executed after and also completed. • Yet you don’t receive speech/audio output from the KB tool even though it produced text output (the logs show tool completion but no speech events for that tool). Most likely cause • Your realtime pipeline only emits or attaches a single speech output per user turn, and the KB tool’s generated reply is not being passed to the TTS/streaming path in the same way as the confirmation reply. In other words, the KB tool produced text, but that text was not routed into the speech generation/streaming step (or it was overwritten by the earlier confirmation speech), so no audio was sent to the client for the KB response. Checks to run now 1. Confirm that

access_knowledge_base

actually calls your session speech/tts method and that call is awaited and completes: ◦ You call

await self.session.generate_reply(instructions=knowledge_instruction)

— ensure

generate_reply

is configured to produce speech in realtime and that it returns/awaits successfully every time. 1. Verify how your session handles multiple generate_reply calls in one turn: ◦ Does

generate_reply

emit audio events immediately, or does it buffer and only send the latest reply’s audio? If it buffers/overwrites, the confirmation’s speech may have reserved the output channel, causing the KB audio to be dropped. 1. Inspect speech IDs and streaming events: ◦ In your logs the confirm tool had

speech_id: "speech_7919f908da86"

, and subsequent entries reference the same speech_id for the KB tool execution. If the KB tool is reusing the same speech_id, the second tool’s audio might be overwriting or ignored by the client. ◦ Confirm whether each call to

generate_reply

should create a new speech_id (and event) rather than reusing one. 1. Check client behavior on multiple audio events: ◦ Ensure the client/player handles receiving back-to-back audio messages in one turn. Some clients only play the first audio chunk unless explicitly instructed to stream successive audio parts or handle multiple speech events. Immediate fixes you can try • Ensure distinct speech events per tool invocation: ◦ Make each

generate_reply

call create its own speech_id / audio event so the client receives both parts. If your session API accepts an explicit

speech_id

or a flag to stream immediately, use it. • Orchestrate outputs into a single composed reply (recommended for deterministic behavior): ◦ Have your router/orchestrator call the confirm tool first, collect its text, then call the KB tool and collect its text, then send one composed

generate_reply

that contains: ▪︎ A confirmation sentence followed immediately by the KB answer. ◦ This guarantees one speech generation call and avoids audio overwrites. • If you need streaming behavior (confirmation speech first, then KB speech): ◦ Send the confirmation via

generate_reply

and await its streaming completion event before calling

generate_reply

for the KB output. If your realtime provider doesn’t expose a completion event, simulate ordering by composing a single reply as above or by sending explicit start/stop markers the client can respect. If you’d like, I can: • Provide a short code snippet that composes both outputs into one

generate_reply

call so audio is always produced. • Show how to ensure

generate_reply

produces separate speech events (if your session API supports creating new speech IDs per call). Which would help more right now — a composed-single-reply snippet to guarantee audio, or a checklist/snippet to ensure distinct speech events are created and streamed per tool?

my flow in this case can be different in this case i have 2 tools

@function_tool()
async def confirm_application_details(self, context: RunContext[SessionState]) -> str:
    """Mark application details as confirmed by customer."""
    self.application_confirmed = True
    context.userdata.call_outcome.application_confirmed = True
    return "Application confirmed."

@function_tool()
async def access_knowledge_base(self, context: RunContext[SessionState], question: str) -> None:
    """Answer customer questions about company services, loans, or financial products.

    Use this tool when the customer asks questions about:
    - Loan products, rates, terms, or conditions
    - Total balance, total payback amount, or total payment
    - Monthly payment amount
    - Approved loan amount or pre-approved amount
    - Company services and offerings
    - Account information or loan details
    - Branch locations or contact information
    - Any finance or lending related topics

    Args:
        question: The customer's question about company/loan topics
    """
    userdata = context.userdata
    company_name = userdata.company_name
    customer_full_name = f"{userdata.customer_data.firstname} {userdata.customer_data.lastname}"
    is_verified = userdata.call_outcome.identity_verified

    general_kb_vars = {
        "servicing_branch_name": userdata.servicing_info.servicing_branch_name,
        "servicing_branch_phone_number": userdata.servicing_info.servicing_branch_phone_number,
    }
    general_kb_content = general_knowledge_base_content.invoke(general_kb_vars).text
    knowledge_content_text = general_kb_content
    if is_verified:
        sensitive_kb_vars = {
            "pre_approved_loan_amount": userdata.loan_info.pre_approved_loan_amount,
            "loan_apr": userdata.loan_info.loan_apr,
            "monthly_payment": userdata.loan_info.monthly_payment,
            "total_payback_amount": userdata.loan_info.total_payback_amount,
            "loan_term_months": userdata.loan_info.terms,
        }
        sensitive_kb_content = sensitive_knowledge_base_content.invoke(sensitive_kb_vars).text
        knowledge_content_text = f"{general_kb_content}\n\n{sensitive_kb_content}"

    knowledge_instruction = base_agent_access_knowledge_base_instructions.invoke(
        {
            "company_name": company_name,
            "customer_full_name": customer_full_name,
            "question": question,
            "knowledge_base_content": knowledge_content_text,
            "servicing_branch_phone_number": general_kb_vars["servicing_branch_phone_number"],
        }
    ).text

    await self.session.generate_reply(instructions=knowledge_instruction)

i found how call them in order, but not i dont get answer from KB, only get let me check

Thanks — this is clear. You’ve confirmed the tools are called in order, but the user only hears “let me check” (the confirmation reply) and not the KB answer. I’ll focus on the exact likely causes and give concrete, minimal changes you can make right now to guarantee the KB answer is delivered (audio/text) while preserving single-turn UX. Quick diagnosis (what’s happening) • Both tools run and complete, but only the confirmation speech/audio is reaching the client. • That indicates the KB tool’s text output is not being routed into the speech/TTS/streaming path (or it’s being overwritten/ignored) after the confirmation audio has already been produced. • Common root causes: ◦ Multiple calls to your session’s

generate_reply

produce separate speech events, but the client or server only plays/keeps the first one. ◦ Both tool calls reuse the same speech ID/channel so the second audio is overwritten or dropped. ◦

generate_reply

for the KB returns text but is not configured to produce audio in realtime (e.g., missing a flag or not awaited to completion). ◦ The realtime stream only allows one active speech output per turn and the earlier confirmation output “wins”. Immediate, guaranteed fix (single composed reply — simplest and most robust) • Don’t call

generate_reply

separately from each tool in the same turn. Instead, collect both tool outputs server-side and call

generate_reply

exactly once with a composed instruction that contains both the confirmation message and the KB answer. This guarantees one speech output that contains both parts. Minimal pattern to implement: 1. Make your tools return their textual outputs instead of calling

generate_reply

directly. ◦ confirm_application_details already returns the confirmation string. ◦ Change access_knowledge_base to return the KB answer text instead of calling

generate_reply

. 1. Create a router/orchestrator (single tool the model calls) that: ◦ Calls

confirm_application_details

→ gets confirm_text. ◦ Calls

access_knowledge_base

→ gets kb_text. ◦ Compose final_reply = f"{confirm_text}\n\n{kb_text}" ◦ Call

await self.session.generate_reply(instructions=final_reply)

once. Why this works: • One

generate_reply

call ensures the realtime TTS/streaming system outputs the combined content as a single audio event; no race/overwrite issues and ordering is guaranteed. Example sketch (small changes only) • change KB tool to return text:

@function_tool()
async def access_knowledge_base(self, context: RunContext[SessionState], question: str) -> str:
    # ... build knowledge_instruction and knowledge_content_text as before ...
    # Instead of generate_reply here, return the instruction text for the orchestrator to send
    return knowledge_instruction

• router tool (the single tool the LLM calls for ambiguous single utterances):

@function_tool()
async def handle_user_message(self, context: RunContext[SessionState], user_text: str) -> None:
    # detect intents (you already have this)
    confirm_text = ""
    kb_text = ""

    # call confirmation if needed
    if detect_confirmation(user_text):
        confirm_text = await self.confirm_application_details(context)

    # call KB if needed (extract question)
    question = extract_question(user_text)
    if question:
        kb_text = await self.access_knowledge_base(context, question)

    # compose single reply and send once
    final_reply = confirm_text
    if kb_text:
        # if kb_text is already formatted instructions, use it directly; otherwise combine
        final_reply = f"{confirm_text}\n\n{kb_text}" if confirm_text else kb_text

    await self.session.generate_reply(instructions=final_reply)

Other approaches if you cannot change tools to return text • If you cannot modify

access_knowledge_base

to return text (because other flows rely on it calling

generate_reply

): ◦ Make

access_knowledge_base

accept an optional flag

suppress_speech: bool = False

. When True, it returns the KB text instead of calling

generate_reply

. Router calls it with

suppress_speech=True

. ◦ Keep the existing behavior when called directly in other contexts. If you must stream confirmation first, then KB in sequence • You need two distinct speech events where the client supports back-to-back playback: ◦ Ensure each

generate_reply

creates a unique speech_id and is awaited through its streaming/completion lifecycle before calling the next

generate_reply

. ◦ If your realtime provider doesn’t surface a completion event, avoid this pattern — prefer the single composed reply. Checks to validate now • Confirm

access_knowledge_base

can return the answer text instead of calling

generate_reply

. If it currently calls

generate_reply

, that explains why its audio may be dropped. • Inspect speech IDs in logs: each call to

generate_reply

should create a different speech_id. If both tools reuse the same ID, change the call to generate a new ID per call. • Test the composed reply approach locally — it should immediately resolve your missing-KB-audio issue. Next steps I can help with • Provide a ready-to-drop code snippet that replaces `access_knowledge_base`’s

generate_reply

call with a return value and adds the router implementation. • Or, if you prefer streaming confirmation first, give a small pattern to ensure unique speech events and a handshake for the client to play both audios in order. Which do you want next — the composed-reply code snippet (preferred) or the multi-speech-event pattern with unique speech IDs?