possible to implement dynamic few shot for tools?

Thanks for reaching out, Kuniaki. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

if we give an agent so many tools, generally, agent's tool call accuracy will be low, right? so, combined with vector search, based on user's utterance, calling the most related tool "dynamically", this is dynamic few shot prompting.

Thanks — that description makes your intent clear and it’s a good approach. Below I’ll outline a practical architecture and step-by-step pattern you can use to implement dynamic few-shot prompting where vector search picks the most relevant tools/examples for an agent at request time. I’ll also list trade-offs and questions so I can tailor a concrete code sample to your stack. Acknowledgement • You’re right: when an agent has many tools, indiscriminate few-shot examples lower tool-call accuracy. Dynamically selecting a small, highly relevant set of tool examples (and/or the most relevant tool definitions) using vector search is an effective way to improve precision. High-level architecture 1. Tool metadata store ◦ For each tool, keep a short description, example usages (I/O pairs), and keywords/tags. ◦ Precompute embeddings for: tool descriptions, each example, and optionally example outputs. 1. Vector index ◦ Store those embeddings in a vector database (e.g., Pinecone, Milvus, Weaviate, or an in-memory index). ◦ Index both tool-level vectors and example-level vectors so you can retrieve either tools or specific examples. 1. Runtime flow per user utterance a. Encode the user utterance to an embedding. b. Query the vector index for nearest items. ▪︎ Option A: Retrieve top-N tool descriptions (to select which tools are relevant). ▪︎ Option B: Retrieve top-M example usages across all tools (to use as few-shot examples). ▪︎ You can do both: find top tools first, then retrieve examples for those tools. a. Filter/prioritize results ▪︎ Apply heuristics (tool availability, user permissions, domain constraints). ▪︎ Optionally re-rank using a small cross-encoder or lightweight classifier for precision. a. Compose the model prompt ▪︎ Inject selected tool definitions (short) and 2–5 few-shot examples chosen dynamically. ▪︎ Clearly mark how the agent should call tools (e.g., a canonical JSON function-call format). a. Send to the model/agent runtime ▪︎ Let the agent decide whether to call a tool; the prompt examples should demonstrate correct usage. Key implementation details and tips • Use two-level retrieval for best results: ◦ Level 1: Retrieve relevant tools (tool selection). ◦ Level 2: For the selected tools, retrieve the most representative examples (few-shot). • Keep tool descriptions short and canonical so the model can match intent to tools easily. • Prefer example diversity within the few-shots: include 1 ambiguous example and 1 clear example if possible. • Limit tokens: prioritize examples by similarity score, then by recentness or frequency. • Use a re-ranker when precision matters: a small model that scores (utterance, tool) pairs can improve top-1 tool accuracy. • If your model supports function-calling APIs, provide explicit function schemas and use them rather than free-text tool prompts—this reduces hallucination. • Cache retrieval results for similar utterances to reduce latency and cost. • Log retrieval + tool-call outcomes to iteratively improve embeddings, examples, and re-ranking. Prompt composition pattern (conceptual) 1. System instructions: agent role + how to format tool calls. 2. Dynamic tool list: brief definition for each top-k tool selected. 3. Few-shot examples: 2–5 examples dynamically selected (showing user utterance -> tool call -> tool output -> agent response). 4. Current user utterance. Example decision heuristics for selection • Similarity threshold: only consider items with cosine similarity above threshold T. • Top-k by score, then apply rule-based elimination (e.g., exclude deprecated tools). • Weighted scoring: combine embedding score, tool usage frequency, recency, and permission score. Trade-offs and considerations • Retrieval quality depends on embeddings and the coverage/quality of your examples. Poor examples mean poor guidance. • More dynamic retrieval increases latency — solve with caching, async prefetching, or a small re-ranker. • Token limits: if many examples are relevant, you’ll need summarization or prioritization. • Maintenance overhead: you’ll need to keep tool docs and examples curated and updated. Next steps / how I can help you further • Tell me your stack so I can give code samples: ◦ Which vector DB (or none)? ◦ Which embedding model? ◦ Which agent runtime or LLM API (and whether it supports structured function-calls)? • If you want, share an example set of tools (names + 1–2 example usages each) and a sample user utterance; I’ll return: ◦ A retrieval + filtering pseudocode flow. ◦ A concrete prompt template with the dynamic few-shot examples inserted. ◦ Suggestions for thresholds and caching strategies. If you share your stack and a small sample, I’ll produce a focused implementation example you can drop into your system. , with interactive elements

is it possible to implement for LiveKit Agents? seems agent should be assigned tools when agents is initialized. (so this is "static").

Thanks — great question, and you’re right to think about the “static vs dynamic” distinction when designing dynamic few-shot prompting for agents. Short answer • LiveKit Agents do not have to be permanently static: while tools are typically defined at initialization, you can modify an agent’s tools at runtime using

agent.update_tools()

. That lets you add, remove, or replace tools after the agent has been created, enabling the dynamic behavior you described. What that enables for dynamic few-shot • You can run your vector search on the user utterance, pick the most relevant tools and examples, then call

agent.update_tools()

to set the agent’s available toolset for that session/turn. • When tools change at runtime, an

AgentConfigUpdate

is automatically added to the conversation history so the LLM is informed about added/removed tools — this helps the agent adapt without confusing context. • You can also group related tools into toolsets (by ID) and swap entire sets when appropriate, which makes management easier for many tools. How you might implement this (conceptual flow) 1. Encode user utterance → perform vector search to find top-N relevant tools and top-M examples. 2. Optionally re-rank/filter results (permissions, availability, similarity threshold). 3. Build the dynamic tool list (or toolset) for this request. 4. Call: ◦ Add or replace tools:

await agent.update_tools([tool_a, tool_b, ...])

◦ Or incrementally:

await agent.update_tools(agent.tools + [new_tool])

or remove with

agent.tools - [tool_a]

1. Send the user utterance to the agent. The conversation history will include an

AgentConfigUpdate

reflecting the tool changes. Practical tips and trade-offs • Keep tool definitions concise so the LLM can match intent quickly. • Limit the number of active tools per turn to improve tool-call accuracy. • Use two-level retrieval (select tools first, then examples) to reduce prompt size and focus the few-shot examples. • Cache common retrieval results to lower latency. • Log retrieval and tool-call outcomes to iteratively improve selection and thresholds. Questions to tailor a concrete example • Which vector DB and embedding model are you using (or planning to use)? • Do you use function-calling style tools (structured schemas) or free-text tool prompts? • Do you want tools updated every single user turn, per session, or only on certain triggers? • Would you like a code example showing the retrieval →

agent.update_tools()

→ prompt flow in Python? If you tell me your stack and desired trigger strategy (per-turn vs session vs conditional), I’ll provide a concrete implementation pattern and a sample snippet you can drop into your LiveKit Agents workflow. Sources: Function tool definition | LiveKit Documentation | Prebuilt tools | LiveKit Documentation, with interactive elements

great. are there any examples?

Thanks — I’ll show a concise end-to-end example and point to existing examples you can inspect. I’m responding to your request for concrete examples showing how to wire a vector search result into LiveKit Agents’ dynamic tool assignment (so you can do dynamic few-shot/tool-selection per turn). Quick summary • LiveKit Agents let you update an agent’s tools at runtime with

agent.update_tools()

(or

updateTools()

in TypeScript). Use that to apply the results of a vector search and limit active tools/examples per turn. • Below is a minimal Python-style conceptual example that ties a vector DB lookup (pick top tools) to

agent.update_tools()

, then runs the agent for that turn. • I’ve also listed relevant repo examples you can use: dynamic tool creation and survey agent examples. Minimal end-to-end pattern (conceptual Python pseudocode) 1. Do a vector search on the user utterance to find the most relevant tool IDs / example IDs. 2. Build the Tool objects (or look them up from a registry). 3. Call

await agent.update_tools(selected_tools)

. 4. Send the user utterance to the agent; the conversation will receive an AgentConfigUpdate and then the agent will use the updated toolset. Example pseudocode

# 1) Assume you have:
# - embed(text) -> embedding vector
# - vector_db.query(vec, top_k) -> list of {id, score, meta}
# - TOOL_REGISTRY: mapping tool_id -> Tool object (function_tool created earlier)

user_text = "I need to refund my last order"
vec = embed(user_text)

# 2) Retrieve top relevant items (tool descriptions or examples)
hits = vector_db.query(vec, top_k=10)

# 3) Choose top-N tools (dedupe & apply heuristics: permissions, availability)
# Example: take top 3 unique tool ids from hits
selected_tool_ids = []
for h in hits:
    tool_id = h["meta"]["tool_id"]  # store tool_id in vector item metadata
    if tool_id not in selected_tool_ids:
        selected_tool_ids.append(tool_id)
    if len(selected_tool_ids) >= 3:
        break

selected_tools = [TOOL_REGISTRY[t] for t in selected_tool_ids]

# 4) Update the agent's tools at runtime
await agent.update_tools(selected_tools)

# 5) Now ask the agent to handle the utterance
reply = await agent.generate_reply({"instructions": user_text})
# AgentConfigUpdate will be appended to the conversation automatically

Notes and variations • Two-level retrieval: first retrieve top tools, then retrieve top few-shot examples only for those tools. Inject the examples into system/instructions or rely on tool schemas + example-based system prompt updates (you can change

update_instructions()

if you want to add dynamic few-shot examples to the prompt). • If you generate tools dynamically (e.g., many similar tools for profile fields), create them with

function_tool()

factories and either register them in TOOL_REGISTRY or pass them directly to

update_tools()

. • Use re-ranking or a similarity threshold to avoid irrelevant tools. • Limit active tools per turn (2–5) to improve tool-call accuracy. TypeScript / JS equivalent • Use

agent.updateTools(...)

or update the tool context in your agent session with the new toolset (examples in the docs and examples repo mirror the same API concepts). • Example usage patterns and helper functions for dynamic creation exist in the examples repo. Concrete examples and repos to inspect • Dynamic tool creation example (complete script): examples/dynamic_tool_creation.py — shows generating tools at runtime and registering them. • Survey agent (Python) example: examples/survey/survey_agent.py — shows real-world agent workflows that change behavior based on session state. • Shopify voice shopper example: a complex example of dynamic agent/tool behavior and MCP integration. Next steps I can help with • If you tell me your stack I’ll produce runnable code: ◦ Python or TypeScript? ◦ Vector DB (Pinecone, Milvus, Weaviate, FAISS, etc.) and embedding model? ◦ Do you use function-calling (structured schemas) or free-text tool prompts? ◦ When should tools be updated: every turn, per session, or on triggers? • I can also provide a full sample that: ◦ Implements two-level retrieval (tools -> examples), ◦ Updates agent tools, ◦ Optionally updates the system prompt with the few-shot examples, ◦ Includes caching and a suggested similarity threshold. Which language and vector DB would you like the runnable example in? Sources: Function tool definition | LiveKit Documentation | Build a salary negotiation coach with a LemonSlice avatar | LiveKit | Agents and handoffs | LiveKit Documentation | Agent Builder | LiveKit Documentation | External data and RAG | LiveKit Documentation, with interactive elements