how to handle end call for gemini 3.1

Thanks for reaching out, Utkarsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

does a single long prompt of 6 pages consume more tokens or a short prompt of 2 pages with 15 tools?

You're asking whether a single long prompt (6 pages) or a shorter prompt (2 pages) plus 15 tool definitions uses more tokens. Short answer: it depends — but often the short prompt with many tools can use equal or more tokens. Why: • Tokens scale with text sent to the model. A 6‑page prompt is large and costs many tokens every call. • Each tool adds structured metadata (name, description, parameter schema) that is also sent as context. Fifteen tools can add substantial token overhead. • If tool schemas are verbose or complex, their combined size may exceed the extra text in the 6‑page prompt. Recommendations: • Measure actual token usage for your specific prompt and tool schemas to be sure. • If you must use many tools, use dynamic loading/caching (e.g., a tool proxy or toolset pattern) so you don't send all tool schemas every request. • Keep system/instruction text concise and avoid redundant details to reduce tokens. If you want, share samples of the 6‑page prompt and one or two tool schemas and I’ll estimate which will consume more tokens and give specific optimizations. Sources: Understand and Improve Agent Latency | LiveKit | Tool definition and use | LiveKit Documentation | Mistral AI LLM plugin guide | LiveKit Documentation | Toolsets | LiveKit Documentation | Groq STT plugin guide | LiveKit Documentation, with interactive elements

give exmaple of dynamic loading and caching

Thanks — you asked for an example of dynamic loading and caching for tools. Below are two concise patterns with code: (A) use ToolProxyToolset to discover and call tools on demand, and (B) dynamically create/update tools at runtime. I also include a simple caching example for pre-synthesized TTS audio. A. ToolProxyToolset (dynamic discovery + minimal schema sent)

from livekit.agents import Agent, function_tool
from livekit.agents.beta.toolsets import ToolProxyToolset, Toolset

@function_tool()
async def get_weather(ctx, location: str) -> str:
    return f"Sunny, 72F in {location}"

class WeatherToolset(Toolset):
    def __init__(self):
        super().__init__(id="weather")
        self._tools = [get_weather]

class TravelAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="You are a travel assistant. Use tool_search to find tools.",
            tools=[
                ToolProxyToolset(
                    id="travel_tools",
                    tools=[WeatherToolset()],  # register toolsets but don't dump all schemas each turn
                    max_results=3,
                )
            ],
        )

• Benefit: The model uses `tool_search`/`call_tool` to discover and invoke only relevant tools, reducing tokens compared to sending all 15 schemas every turn. B. Dynamic creation / update of tools at runtime

from livekit.agents import Agent, function_tool

def create_db_tool(table: str, op: str):
    schema = {
        "name": f"{op}_{table}",
        "description": f"{op} on {table}",
        "parameters": { "type": "object", "properties": { "id": {"type": "string"} }, "required": ["id"] }
    }
    async def handler(raw_args, ctx):
        return f"Performed {op} on {table} id={raw_args['id']}"
    return function_tool(handler, raw_schema=schema)

class DBAgent(Agent):
    async def add_user_tools(self, tables_ops):
        tools = [create_db_tool(t, o) for t,o in tables_ops]
        await self.update_tools(tools)

• Benefit: Only the necessary tools are added to the agent when needed, so you avoid sending unused tool schemas. C. Simple caching example (pre-synthesize and reuse TTS audio)

HOLD_FRAMES = []

async def preload_hold_message(tts):
    global HOLD_FRAMES
    async for event in tts.synthesize("Let me check that for you."):
        HOLD_FRAMES.append(event.frame)

@function_tool()
async def long_lookup(ctx, order_id: str) -> str:
    # play cached audio while doing work
    async def cached_audio():
        for f in HOLD_FRAMES:
            yield f
    hold_handle = ctx.session.say("Please wait...", audio=cached_audio(), add_to_chat_ctx=False)
    result = await fetch_order(order_id)
    if not hold_handle.done(): hold_handle.interrupt()
    return result

• Benefit: Avoid repeated TTS token/latency for fixed phrases. Practical tips • Prefer ToolProxyToolset or ToolSearchToolset when you have many tools. • Keep tool schemas compact (short descriptions, minimal parameter metadata). • Measure token usage and latency after implementing; iterate (cache heavy schemas or use runtime loading if needed). If you want, paste one or two tool schemas and I’ll show a concrete before/after token estimate and a tailored ToolProxy setup. Sources: Toolsets | LiveKit Documentation | Function tool definition | LiveKit Documentation | Audio customization | LiveKit Documentation | The ReAct Pattern for Voice Agents and How AI Agents Think, Act, and Respon..., with interactive elements

which tools are take into consideration for calculating the total token for a voice call

You're asking which tools count toward total token usage for a voice call. Short answer: any tool whose schema, invocation, or returned data is sent to or processed by the LLM contributes to tokens. What counts • System/user/assistant prompt text (every turn). • Tool definitions sent in context (name, description, parameter schema). • Tool invocation arguments sent to the model. • Tool responses returned to the model (tool output text/JSON). • STT transcripts fed into the LLM as user messages. • LLM outputs (responses) and any follow-up messages. • Provider tool calls (e.g., web search, code interpreter) — their input/output exchanged with the model. • Cached vs. uncached tokens are tracked per message (input/output/cached). Practical notes • Large or many tool schemas increase input tokens even if not invoked. • Tool outputs can be large and add substantially to total tokens. • Use dynamic tool loading (ToolProxyToolset/ToolSearchToolset) or update_tools at runtime to avoid sending all schemas every turn. • LiveKit exposes per-turn token metrics (input/output/total) so you can measure which tools or turns drive cost. If you share your tool schemas or a sample call flow, I can point out which specific parts will add the most tokens and suggest optimizations. Sources: The ReAct Pattern for Voice Agents and How AI Agents Think, Act, and Respon... | Tool definition and use | LiveKit Documentation | Understand and Improve Agent Latency | LiveKit | Understand and Improve Agent Latency | LiveKit, with interactive elements

what tokens are taken into account for total_tokens calculations?

You're asking exactly which token types are included in the

total_tokens

calculation. Here's a concise breakdown. What counts toward total_tokens • Input tokens: ◦ System/user/assistant prompt text per turn. ◦ STT transcripts (transcribed user speech sent as text). ◦ Tool schemas/function definitions sent in context. ◦ Tool invocation arguments passed to the LLM. ◦ Inputs sent to provider tools (e.g., web search) when those are routed through the LLM. ◦ Cached input tokens are tracked separately as

cached_tokens

but factor into metrics and efficiency. • Output tokens: ◦ LLM-generated responses (text or structured outputs). ◦ Text sent to TTS (unless using a realtime model where audio/text may be bundled). • Tool-related tokens: ◦ Tool results/outputs returned to the LLM (JSON/text) count as tokens. ◦ Provider tool input/output (e.g., CodeInterpreter, WebSearch) are included. Notes • For realtime models, metrics break down into

input_tokens

(including

audio_tokens

for STT,

text_tokens

,

cached_tokens

) and `output_tokens`;

total_tokens

is the sum of those. • Large tool schemas or many tools increase input tokens even when not invoked. • LiveKit exposes per-turn token metrics (

RealtimeModelMetrics

/ per-message metrics) so you can see input/output/cached token breakdowns for optimization. If you want, share a sample turn (prompt + a tool schema + a tool response) and I’ll show a concrete token breakdown. Sources: Mistral AI LLM plugin guide | LiveKit Documentation | https://livekit.com/pricing.md| | Agent Console | LiveKit Documentation, with interactive elements