https://livekit.io logo
Join Slack
Powered by
how to load tools using lazy loading
# ask-ai
p
how to load tools using lazy loading
✅ 1
t
Thanks for reaching out, arpan. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about how to load tools using lazy loading. LiveKit provides dynamic tool discovery for lazy loading via two Python-only beta toolsets: • ToolSearchToolset — exposes `tool_search`; when the LLM calls it, matching tools are added to the model’s tool list for the next turn. • ToolProxyToolset — exposes 
tool_search
and `call_tool`; keeps a fixed tool list so providers can reuse prompt caches while still loading tool implementations on demand. Quick example (Python):
Copy code
from livekit.agents.beta.toolsets import ToolProxyToolset

class TravelAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="You are a travel planning assistant. Use tool_search to find the right tools.",
            tools=[
                ToolProxyToolset(
                    id="travel_tools",
                    tools=[
                        WeatherToolset(),
                        FlightToolset(),
                        convert_currency,
                    ],
                    max_results=3,
                )
            ],
        )
Notes: • Default search uses BM25 ranking; you can switch to 
KeywordSearchStrategy
or implement a custom 
SearchStrategy
. • Supported lazy-loaded items: standalone function tools, 
Toolset
groups, and 
MCPToolset
servers. • Feature is Python-only and marked Beta. Do you want an example using 
ToolSearchToolset
, a custom search strategy, or help integrating this into your existing agent code? Sources: Function tool definition | LiveKit Documentation, with interactive elements
p
yes
t
Great — which of these would you like next? • An expanded example using ToolSearchToolset (adds tools to the model's tool list when searched). • A deeper example using ToolProxyToolset with 
call_tool
(keeps a fixed proxy list and invokes tools on demand). • How to implement a custom SearchStrategy (replace BM25 with your own ranking). • Help integrating lazy loading into your existing agent code — if so, please share a short snippet or describe your agent setup (Python version, agent framework/code sample)., with interactive elements
p
one problem iam facing is that models response during conversation it taking 2 to 3s my llm avfe 560ms why this latency comming
t
Thanks — I’m looking into why you see 2–3s end-to-end latency while your LLM inference is ~560ms. A few quick questions to pinpoint the cause: 1. Which model provider and API are you using (OpenAI, Anthropic, local LLM, etc.)? 2. Are you using streaming responses or non‑streaming (full response returned at once)? 3. Do you call any tools, `tool_search`/`call_tool`, or external APIs during the turn? If so, which and how long do those calls take? 4. Is your agent using 
ToolProxyToolset
/ 
ToolSearchToolset
or a fixed tool list? 5. Where is your agent/runtime hosted vs. the model provider (same region or across regions)? Any network latency numbers (ping/traceroute)? 6. Do you see delays before inference starts (request queued) or only after inference completes? Any timestamps/logs you can share for a single request (send time, model start, model end, response received)? 7. Are you doing any post‑processing (serialization, reranking, embeddings search) after model output? Based on common causes, likely contributors are: • Non‑streaming response buffering on the provider side. • Tool discovery or external API calls invoked during the turn. • Network latency between your runtime and the model provider. • Prompt construction, retrieval/embedding lookups, or reranking work done before/after inference. • Prompt cache misses or repeated context prep each turn. Share answers (or a short request/response timeline) and I’ll suggest targeted checks and fixes. , with interactive elements
p
llm OpenAI stt servam tts certsea
t
Thanks — I’m looking into the 2–3s end-to-end latency when your LLM (OpenAI) inference is ~560ms and you’re also using STT and TTS. I’ll help narrow the cause — a few targeted questions and checks will point us to the biggest contributors. Quick clarifying questions: 1. Are you using streaming for the LLM responses, STT, or TTS, or are those non‑streaming (full response returned at once)? 2. Which OpenAI API endpoint/model are you calling (e.g., gpt-4o, gpt-4.1, gpt-4o-mini) and via which SDK (openai HTTP, official client, or proxy)? 3. Do you call tools (`tool_search`/`call_tool`) or external APIs during the turn (including knowledge retrieval or embeddings lookups)? If yes, which calls and approximate durations? 4. Where is your agent/runtime hosted (region/VM provider) relative to the OpenAI region? Any network RTTs or ping numbers? 5. Are STT and TTS external services or OpenAI speech endpoints? What are their average latencies when called alone? 6. Do you observe delay before the model starts generating (queue/latency) or between model finish and when audio/playback is produced? Can you share timestamps for: request sent, model start, model end, STT/TTS start/end, response returned? 7. Are you doing local processing (prompt building, embedding similarity search, re‑ranking) before or after the model call? If so, how long do those steps take? Quick checks you can run now: • Measure and log per-step timings (timestamp each: STT, prompt construction, model request start, model response streaming chunks, tool calls, TTS start/end) for a single request. • Test model call in isolation from the same host to confirm 560ms inference is consistent end-to-end (no extra network/proxy overhead). • If using non‑streaming model calls, try streaming to see if first-byte latency improves perceived responsiveness. • If using ToolProxyToolset/ToolSearchToolset, disable tool calls temporarily to see impact. • Check network RTT to OpenAI endpoints from your runtime (curl with --trace-time or ping/traceroute). If you share one sample timing log (timestamps or durations per step) or answer 1–4, I’ll recommend specific optimizations (e.g., enable streaming, move services to same region, cache prompt context, use ToolProxy to avoid prompt cache misses, parallelize TTS generation). , with interactive elements
Open in Slack
PreviousNext
Powered by