I'm retrieving product catalog with a tool call, and I'm getting quite high e2e latency. If normal requests are 1s, with tool calling it's 3+s. How's it technically working, does the llm return the response that it needs to make a tool call, the tool call is made, and another llm request with the data is sent, that's why I get so significant difference? or I'm getting it wrong?

Thanks for reaching out, Alexandru. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Yes, the answer is simple and determenistic. Is it technically possible to have first llm inference return the template of the response to which the tool call answer is appended and sent to tts directly? Is it something done in other setups? is this what you offered?

Thanks — that’s a great follow-up. I’m replying to your question about whether the first LLM pass can produce a response template that you then fill with the tool result and send straight to TTS (skipping the second LLM call), whether other setups use this, and whether it aligns with the optimizations suggested. Short answer • Yes — it’s technically possible to have the first LLM inference output a response template, then append the tool result to that template and send it directly to TTS without a second LLM inference. Many teams use this pattern when the final answer is deterministic and templated to reduce latency. How it typically works in this pattern 1. First LLM inference returns: ◦ A structured decision (tool name + args), and ◦ A response template (e.g., "I found X items. The top result is: {item_snippet}. Would you like more details?") 1. Your runtime calls the tool and receives the result (e.g., product details or list). 2. Your application fills the template placeholders with the tool output (formatting/truncation as needed). 3. The completed text is sent directly to TTS and/or returned to the user — no second LLM call required. Benefits • Eliminates the second LLM inference latency (often a significant savings). • Predictable, deterministic output that you can control/validate before TTS. • Easier to cache and audit (since the template is fixed and only data changes). Trade-offs and caveats • Loss of dynamic natural-language polishing: you won’t get the LLM’s ability to rewrite, summarize, or adapt phrasing based on the exact tool result. The output will be as good as your template and any formatting logic you implement. • Template brittleness: if tool outputs vary widely, you must handle edge cases (missing fields, long text, special characters) in your templating code. • Safety/validation: you need to sanitize and possibly filter tool results before sending to TTS (e.g., remove PII or malformed content). • Complex composition: for multi-step or deeply contextual responses, a second LLM pass can produce more natural, cohesive language. Templates work best when responses are short and structured. Is this commonly done? • Yes — teams frequently use templated-first-pass + fill-once for cases where: ◦ Responses are short and structured (product summaries, status checks, simple lookups). ◦ Latency matters (voice agents, IVR, or real-time UX). ◦ Determinism and safety are priorities. • For richer, conversational replies, teams often keep the second LLM pass for better language quality. Implementation tips • Keep templates minimal and include placeholder length limits to prevent overly long TTS output. • Validate and sanitize tool data before insertion into the template. • Use short templates for TTS to improve clarity and reduce synthesis time. • Add a fallback: if the tool returns an unexpected shape or too-large payload, optionally trigger the second LLM pass only in that case. • Measure quality vs latency: run A/B tests comparing template-only responses with full two-pass responses to see if users notice the language difference. If you want, I can: • Help design a safe, compact template for your product responses. • Suggest formatting/validation rules for product fields (titles, price, descriptions) before TTS. • Propose a fallback decision flow that calls the second LLM only when needed. Which would be most useful: a sample template and validation checklist, or a fallback decision flow?

checked the tool call responses now, not so deterministic as I thought. What are other options to decrease latency safely

Thanks — I hear you: your tool outputs are non‑deterministic, so the template-first approach isn’t reliable. Below are focused, practical options you can apply to reduce end-to-end latency safely while preserving correctness. Acknowledgement • You said tool responses aren’t deterministic, so we should avoid approaches that assume fixed output. I’ll prioritize options that keep correctness and safety while lowering latency. High-impact options (safe with non-deterministic tools) 1. Reduce network distance (biggest win) ◦ Co-locate your agent runtime and inference/TTS/STT services in the same region or data center to cut round‑trip times. ◦ Where possible use private networking or provider regions that are closest to your backend. 1. Use streaming everywhere ◦ Stream STT so transcription becomes available as speech arrives. ◦ Stream LLM tokens so you can start TTS (or send partial audio) before the full response finishes. ◦ Stream TTS synthesis to start audio playback while later parts are still generating. ◦ Streaming reduces perceived latency and allows overlapping stages. 1. Parallelize safe parts of the pipeline ◦ While waiting for tool results, start non-dependent work: e.g., prepare static UI elements, load voice assets, or begin TTS for known intro text ("One moment, searching..."). ◦ Preload voices and connection handshakes so TTS starts immediately when text is ready. 1. Preemptive/speculative generation (cancellable) ◦ Speculatively generate likely replies while the user is still speaking. If the tool result diverges, cancel or discard the speculation. ◦ This can safely reduce latency when there’s a high probability of a particular follow‑up. 1. Short, robust templates + selective LLM rerun ◦ Use short templates for predictable parts (e.g., status messages or confirmations), but only fill tool data after validating it. ◦ If the tool output is unexpected or needs polishing, then run the second LLM pass. In other words, route the common fast path to template + TTS, and fallback to two‑pass only when necessary. 1. Cache aggressively but safely ◦ Cache recent or frequently requested product lookups; use TTLs and validate freshness as needed. ◦ For slightly stale but acceptable content, serve cached data immediately and refresh in the background. ◦ Always include freshness metadata to avoid serving stale critical info (prices, availability). 1. Add UX mitigations to improve perceived latency ◦ Play a short “thinking” sound or voice: reduces user frustration while the tool call completes. ◦ Provide brief verbal status updates for long operations (“I’m pulling the latest product info, this will take a moment”). ◦ These don’t reduce actual latency but improve user experience when variability exists. 1. Ensure robust validation & sanitization ◦ Before sending tool results to TTS or the user, validate shapes, truncate long fields, and remove or mask sensitive content. ◦ If validation fails, call the second LLM pass or return a safe fallback message. 1. Choose lower-latency providers and optimize model choice ◦ For latency-sensitive stages, use faster LLMs or lower‑latency providers. ◦ Use smaller/faster models for the decisioning step and reserve larger models only where quality is required. 1. Instrument and measure ◦ Break down latency into first LLM decision, tool call time, second LLM (if any), and TTS. Track TTFT and TTFB. ◦ Use these measurements to target the real bottleneck rather than guessing. Minimal-change flow you can implement quickly 1. Keep the two‑pass flow, but: ◦ Add a short templated intro (pre-TTS) to play immediately while tool call runs. ◦ Start streaming LLM output and stream TTS as soon as the first tokens are available. 1. Add a validation step on tool output: ◦ If tool output matches expected shape and size, fill template and send to TTS. ◦ Else, trigger the second LLM pass to polish and produce the final text. When to still use the two-pass approach • Use the second LLM pass when tool output needs summarization, safety checks, or rephrasing for conversational quality. Don’t eliminate it when the tool output’s variability could harm correctness or safety. If you want, I can: • Draft a compact validation checklist for product fields (title, price, availability, description) to decide when to skip the second LLM. • Propose a short template + fallback decision flow you can drop into your runtime. Which would help you most right now? Sources: How can I reduce latency in voice agents using STT, TTS and LLM? | LiveKit | examples/drive-thru | livekit/livekit_composite | DeepWiki | examples/frontdesk | livekit/livekit_composite | DeepWiki | Introducing LiveKit Inference: A unified model interface for voice AI | Introducing LiveKit Inference: A unified model interface for voice AI