which llm metrics reflect correct utilisation of t...
# ask-aib
boundless-author-7811
02/26/2026, 12:32 PMwhich llm metrics reflect correct utilisation of tokens/minute?
t
tall-airline-16756
02/26/2026, 12:32 PMThanks for reaching out, Utkarsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
tall-airline-16756
02/26/2026, 12:32 PMThanks — you're asking which LLM metrics show correct utilization of tokens per minute. Good question.
The key metrics to monitor are:
• Prompt Tokens — tokens in the input you send to the model.
• Completion Tokens — tokens generated by the model in the response.
• Total Tokens — sum of prompt + completion tokens for a single request; use this to measure actual token consumption.
• Tokens/Second — processing throughput for requests; useful to understand how quickly tokens are being consumed.
These metrics are emitted via the
LLMMetricsTotal TokensTokens/SecondLLMMetricsb
boundless-author-7811
02/26/2026, 1:08 PMis the above correct for realtime models as well?
t
tall-airline-16756
02/26/2026, 1:09 PMThanks — you're asking whether the same LLM token metrics (Prompt Tokens, Completion Tokens, Total Tokens, Tokens/Second) apply to real-time models. Short answer: yes.
What applies to real-time models
• Prompt Tokens, Completion Tokens, Total Tokens are emitted for real-time models just like for standard LLMs.
• Tokens/Second (throughput) is also available and useful for spotting performance changes in real-time streams.
• LiveKit Agents emit these values via the
LLMMetricsmetrics_collectedtotal_tokensmetrics_collectedb
boundless-author-7811
02/26/2026, 1:09 PMprompt tokens are constant or recalculated everytime?
t
tall-airline-16756
02/26/2026, 1:10 PMThanks — you're asking whether
prompt_tokensprompt_tokensLLMMetricsmetrics_collectedtotal_tokensprompt_tokensprompt_tokensb
boundless-author-7811
02/26/2026, 1:19 PM{'call_start_time': 1772005831.0120091, 'call_end_time': 1772005985.8791935, 'llm_prompt_tokens': 243899, 'llm_completion_tokens': 6633, 'total_tokens': 250532}, is this log correct?
t
tall-airline-16756
02/26/2026, 1:20 PMThanks — you asked whether this log is correct. I checked the numbers and have a short analysis and next steps.
Quick checks
• Call duration: 1772005985.8791935 − 1772005831.0120091 = ~154.87 seconds (~2 minutes 35 seconds) — plausible.
• Token arithmetic: 243,899 (prompt) + 6,633 (completion) = 250,532 (total) — arithmetic is correct.
Where this looks suspicious
•
llm_prompt_tokens = 243,899llm_completion_tokens = 6,633LLMMetricsmetrics_collectedb
boundless-author-7811
02/26/2026, 1:42 PMsummary = usage_collector.get_summary()
usage_data = {
'llm_prompt_tokens': summary.llm_prompt_tokens,
'llm_completion_tokens': summary.llm_completion_tokens,
'total_tokens': summary.llm_prompt_tokens + summary.llm_completion_tokens,
}, i am using this process
t
tall-airline-16756
02/26/2026, 1:42 PMThanks — you shared the code you’re using to collect usage and asked whether that process is correct. I’ll confirm what’s happening and point out the likely reason you saw the very large prompt token count.
What your code does
•
summary = usage_collector.get_summary()llm_prompt_tokens: summary.llm_prompt_tokensllm_completion_tokens: summary.llm_completion_tokenstotal_tokensllm_prompt_tokensllm_prompt_tokensLLMMetricsmetrics_collectedget_summary()metrics_collectedtotal_tokensmetrics_collectedget_summary()b
boundless-author-7811
02/27/2026, 5:16 AM Copy code
async def _create_agent_and_session(self):
<http://self.logger.info|self.logger.info>("[%s] 🤖 Starting customer session...", self.room_name)
# Get audio configuration for noisy phone environments
audio_config = self.audio_config_service.get_noisy_environment_config()
# Create Silero VAD for local speech detection (filters noise before OpenAI)
self.silero_vad = self.audio_config_service.create_silero_vad(audio_config)
# Get turn detection config for OpenAI semantic VAD
turn_detection = self.audio_config_service.get_turn_detection_config(audio_config)
# Create Model with semantic VAD for noise resilience
model = self.model_config.create_model(self.agent_config, turn_detection)
# Create agent - tools come from agent_config.tools (ToolSchema objects)
agent = GenericAgent(
agent_config=self.agent_config,
customer_data=self.customer_data,
on_transfer_requested=None # PHASE 2: Supervisor transfer disabled
)
# Load dynamic tools (agent handles this internally)
await agent.initialize_dynamic_tools()
<http://self.logger.info|self.logger.info>("✅ Agent ready with %d tools", len(agent.tools))
# Create Session with Silero VAD and tuned interruption settings
self.agent_session = AgentSession(
llm=model,
vad=self.silero_vad, # Silero VAD for noise filtering
min_interruption_duration=audio_config.min_interruption_duration,
min_interruption_words=audio_config.min_interruption_words,
)
# Setup Metrics Collector
self.usage_collector = metrics.UsageCollector()
@self.agent_session.on("metrics_collected")
def _on_metrics_collected(ev: MetricsCollectedEvent):
metrics.log_metrics(ev.metrics)
self.usage_collector.collect(ev.metrics)
# Store metadata for tools and shutdown
self.agent_session.metadata = {
"sip_trunk_id": self.call_context.trunk_id,
"customer_id": self.customer_data.customer_id,
"customer_name": self.customer_data.customer_name,
"campaign_id": self.customer_data.campaign_id,
"call_context": self.call_context
}
<http://self.logger.info|self.logger.info>(
"[%s] ✅ Agent session created with Silero VAD (threshold=%.2f) and Semantic VAD",
self.room_name,
audio_config.silero.activation_threshold
)
return agent
def get_usage_collector(self) -> Optional[metrics.UsageCollector]:
"""Get the usage collector for metrics reporting"""
return self.usage_collectort
tall-airline-16756
02/27/2026, 5:16 AMThanks — I reviewed your agent/session code, the usage-collector pattern, and the RealtimeModel logs you pasted. I’ll confirm what’s happening, point out what the log fields mean, and show how to compute tokens/minute reliably from your current setup.
What the logs represent
• Each RealtimeModel log line is a per-invocation metrics snapshot emitted by the agent (via
metrics_collectedLLMMetricsinput_tokensinput_text_tokenscached_input_tokensoutput_tokensoutput_text_tokenstimestampjob_id@agent_session.on("metrics_collected")self.usage_collector.collect(ev.metrics)usage_collector.get_summary()total_tokenscached_input_tokens Copy code
s0 = usage_collector.get_summary()
await asyncio.sleep(60)
s1 = usage_collector.get_summary()
prompt_delta = s1.llm_prompt_tokens - s0.llm_prompt_tokens
completion_delta = s1.llm_completion_tokens - s0.llm_completion_tokens
tokens_per_minute = prompt_delta + completion_deltausage_collector.collect()input_tokensb
boundless-author-7811
02/27/2026, 5:18 AMbut at the end i am getting: Feb 26, 2026 @ 184558.439
{"message": "[repayment_pdd-sib-13-1772111640] 📊 Call Metrics: {'call_start_time': 1772111649.6190197, 'call_end_time': 1772111754.533457, 'llm_prompt_tokens': 82245, 'llm_completion_tokens': 2356, 'total_tokens': 84601}", "level": "INFO", "name": "orchestrator.shutdown_handler", "pid": 105, "job_id": "AJ_ieL8Kn4ZvxoB", "room_id": "RM_TK7TeUrb2NVY", "timestamp": "2026-02-26T131558.438605+00:00"}, as you can see for the above logs the: llm_prompt_tokens: 82245 is huge, how this summation is coming from the above logs?
t
tall-airline-16756
02/27/2026, 5:19 AMThanks — I looked at the per‑invocation RealtimeModel logs you pasted and your final call summary. I’ll explain exactly how the 82,245 prompt tokens can legitimately arise from that stream and what to check so you can validate the numbers.
Short answer
• The usage collector sums prompt/input tokens across every invocation it receives. The 82,245
llm_prompt_tokensinput_tokensmetrics_collectedself.usage_collector.collect(ev.metrics)input_tokensinput_tokensllm_prompt_tokenscached_input_tokensinput_tokenssummary.llm_prompt_tokenscollect()metrics_collectedinput_tokensusage_collector.get_summary()b
boundless-author-7811
02/27/2026, 5:26 AM{"message": "RealtimeModel metrics", "level": "INFO", "name": "livekit.agents", "model_name": "gpt-realtime", "model_provider": "fpl-voicebot-realtime.openai.azure.com", "ttft": 0.83, "input_tokens": 7814, "cached_input_tokens": 6272, "input_text_tokens": 6368, "input_cached_text_tokens": 5824, "input_image_tokens": 0, "input_cached_image_tokens": 0, "input_audio_tokens": 1446, "input_cached_audio_tokens": 448, "output_tokens": 76, "output_text_tokens": 22, "output_audio_tokens": 54, "output_image_tokens": 0, "total_tokens": 7890, "tokens_per_second": 52.39, "pid": 105, "job_id": "AJ_ieL8Kn4ZvxoB", "room_id": "RM_TK7TeUrb2NVY", "timestamp": "2026-02-26T131556.347673+00:00"}, how is the input_tokens calculated, it is the summation of which fields?
t
tall-airline-16756
02/27/2026, 5:26 AMThanks — you asked exactly which fields make up
input_tokensinput_text_tokensinput_audio_tokensinput_image_tokensinput_tokensinput_cached_text_tokensinput_cached_audio_tokenscached_input_tokensinput_tokensinput_cached_text_tokensinput_cached_audio_tokenscached_input_tokensinput_tokensinput_tokensmetrics_collectedinput_tokensb
boundless-author-7811
02/27/2026, 5:45 AMwhat about the output tokens?
t
tall-airline-16756
02/27/2026, 5:45 AMGood question — output tokens are calculated the same way input tokens are, but they represent what the model generated (no cached subset). I’ll explain exactly which fields sum to
output_tokensoutput_tokens: 76output_tokensllm_completion_tokensmetrics_collectedinput_tokensllm_completion_tokensoutput_tokensoutput_tokensoutput_tokenssummary.llm_completion_tokensinput_tokensoutput_tokensb
boundless-author-7811
02/27/2026, 5:54 AMis the total_prompt_tokens correct?
t
tall-airline-16756
02/27/2026, 5:55 AMThanks — I’ll confirm whether the session’s reported
llm_prompt_tokensinput_tokensinput_tokensmetrics_collectedinput_tokensusage_collector.collect(ev.metrics)usage_collector.get_summary().llm_prompt_tokensinput_tokensinput_tokens Copy code
# assume logs is a list of parsed JSON log dicts for the session
sum_input = sum(line["input_tokens"] for line in logs)
sum_cached = sum(line.get("cached_input_tokens", 0) for line in logs)
sum_non_cached = sum_input - sum_cached
print("sum_input:", sum_input)
print("sum_cached:", sum_cached)
print("sum_non_cached:", sum_non_cached)input_tokensinput_text_tokenscollect()input_tokensb
boundless-author-7811
02/27/2026, 6:00 AMis the avg token used when doing real time call?
t
tall-airline-16756
02/27/2026, 6:00 AMThanks — you’re asking what the “average token used” is for a real‑time call and how to compute it from your logs. I’ll explain how to calculate relevant averages (per‑call, per‑minute, and per‑invocation), show formulas using fields from your RealtimeModel logs, and suggest which averages are most useful for monitoring and billing.
Key averages you can compute
• Average tokens per invocation (turn)
◦ Useful to understand how large each model request is.
◦ Formula: avg_tokens_per_invocation = (Σ total_tokens_i) / N_invocations
◦ From your logs: sum the
total_tokensinput_tokensoutput_tokenstokens_per_secondtokens_per_secondinput_tokensoutput_tokensusage_collector.get_summary()b
boundless-author-7811
02/27/2026, 6:06 AMno, i am asking if the total tokens are correct for real time communication?
t
tall-airline-16756
02/27/2026, 6:07 AMThanks — you’re asking whether the reported total tokens (and prompt tokens) are correct for real‑time communication. Short answer: yes — the totals look correct given your logs and how the collector works. Here’s why, exactly what’s being summed, and how you can verify it.
Why the large totals are expected
• Each RealtimeModel log line is a single invocation’s metrics snapshot. For each invocation:
◦ input_tokens = input_text_tokens + input_audio_tokens + input_image_tokens
◦ output_tokens = output_text_tokens + output_audio_tokens + output_image_tokens
◦ total_tokens = input_tokens + output_tokens
• Your usage collector receives every
metrics_collectedinput_tokenscached_input_tokensinput_cached_*input_tokenscollect()metrics_collected Copy code
sum_input = sum(line["input_tokens"] for line in logs_for_session)
sum_output = sum(line["output_tokens"] for line in logs_for_session)
sum_total = sum_input + sum_output
# Compare sum_input to summary.llm_prompt_tokens, sum_output to summary.llm_completion_tokens Copy code
sum_billable_prompt = sum(line["input_tokens"] - line.get("cached_input_tokens", 0) for line in logs_for_session)input_text_tokensinput_cached_audio_tokensinput_tokensoutput_tokensb
boundless-author-7811
03/04/2026, 9:00 AM Copy code
async def _create_agent_and_session(self):
<http://self.logger.info|self.logger.info>("[%s] 🤖 Starting customer session...", self.room_name)
# Get audio configuration for noisy phone environments
audio_config = self.audio_config_service.get_noisy_environment_config()
# Create Silero VAD for local speech detection (filters noise before OpenAI)
self.silero_vad = self.audio_config_service.create_silero_vad(audio_config)
# Get turn detection config for OpenAI semantic VAD
turn_detection = self.audio_config_service.get_turn_detection_config(audio_config)
# Create Model with semantic VAD for noise resilience
model = self.model_config.create_model(self.agent_config, turn_detection)
# Create agent - tools come from agent_config.tools (ToolSchema objects)
agent = GenericAgent(
agent_config=self.agent_config,
customer_data=self.customer_data,
on_transfer_requested=None # PHASE 2: Supervisor transfer disabled
)
# Load dynamic tools (agent handles this internally)
await agent.initialize_dynamic_tools()
<http://self.logger.info|self.logger.info>("✅ Agent ready with %d tools", len(agent.tools))
# Create Session with Silero VAD and tuned interruption settings
self.agent_session = AgentSession(
llm=model,
vad=self.silero_vad, # Silero VAD for noise filtering
min_interruption_duration=audio_config.min_interruption_duration,
min_interruption_words=audio_config.min_interruption_words,
)
# Setup Metrics Collector
self.usage_collector = metrics.UsageCollector()
@self.agent_session.on("metrics_collected")
def _on_metrics_collected(ev: MetricsCollectedEvent):
metrics.log_metrics(ev.metrics)
self.usage_collector.collect(ev.metrics)
# Store metadata for tools and shutdown
self.agent_session.metadata = {
"sip_trunk_id": self.call_context.trunk_id,
"customer_id": self.customer_data.customer_id,
"customer_name": self.customer_data.customer_name,
"campaign_id": self.customer_data.campaign_id,
"call_context": self.call_context
}
<http://self.logger.info|self.logger.info>(
"[%s] ✅ Agent session created with Silero VAD (threshold=%.2f) and Semantic VAD",
self.room_name,
audio_config.silero.activation_threshold
)
return agent
def get_usage_collector(self) -> Optional[metrics.UsageCollector]:
"""Get the usage collector for metrics reporting"""
return self.usage_collectort
tall-airline-16756
03/04/2026, 9:00 AMGreat — you’re seeing high summed
input_tokensinput_tokensinput_tokenscached_input_tokenscached_input_tokensmetrics_collected Copy code
class TokenTracker:
def __init__(self):
self.prev_input = 0
self.prev_output = 0
def on_metrics(self, m):
delta_input = m.input_tokens - self.prev_input
delta_output = m.output_tokens - self.prev_output
# billable input for this turn
cached = getattr(m, "cached_input_tokens", 0)
billable_input = delta_input - cached if delta_input > cached else 0
self.prev_input = m.input_tokens
self.prev_output = m.output_tokens
return delta_input, delta_output, billable_input Copy code
s0 = usage_collector.get_summary()
await asyncio.sleep(60)
s1 = usage_collector.get_summary()
tokens_last_min = (s1.llm_prompt_tokens + s1.llm_completion_tokens) - (s0.llm_prompt_tokens + s0.llm_completion_tokens)input_tokensb
boundless-author-7811
03/04/2026, 9:06 AMcan multi agent approach solve this issue? Let say I have will have initial_greeting_agent, then name_confirmation_agent and then other agents so on based on condition, as the converstaion proceeds the agents gets loaded. Does this approach makes sense, if so, how should we do it?
t
tall-airline-16756
03/04/2026, 9:06 AMShort answer: Yes — a multi‑agent approach can reduce token usage if designed correctly. It’s a good fit when different conversation phases need different system prompts, toolsets, or context. Below I’ll explain why it helps, trade‑offs, and a practical way to implement it with your LiveKit/AgentSession pattern.
Why this helps
• Smaller per‑agent system prompts and tool definitions mean less context sent each invocation. That reduces the repeated token cost when that agent is active.
• You can keep each agent’s conversation history short (only relevant turns), avoiding one ever‑growing global context.
• You can load heavy tools or long instructions only when needed and unload them afterwards, lowering steady-state input tokens.
Trade-offs to consider
• Context continuity: switching agents can lose fine-grained recent context unless you intentionally hand off a summary or the necessary state.
• Complexity: more agent instances and routing logic increases implementation and testing effort.
• Latency/overhead: creating/initializing agents frequently may add small delays unless you reuse warmed instances.
How to design it (practical pattern)
1. Define clear responsibilities
◦ initial_greeting_agent: short greeting prompt, small toolset (e.g., IVR menu)
◦ name_confirmation_agent: focused prompt for confirming identity
◦ verification_agent / billing_agent / fallback_agent: each with minimal instructions and only required tools
1. Use lightweight system prompts and minimal tool schemas per agent
◦ Keep system messages and tool descriptions concise.
◦ Move large static references to an external retrieval service instead of embedding in the prompt.
1. Hand off context explicitly
◦ When switching agents, pass only what the next agent needs:
▪︎ Option A: send a brief structured handoff object (e.g., {customer_name, verified: true, last_user_utterance})
▪︎ Option B: send a short summary generated at handoff time:
▪︎ Summarize recent turns to 1–2 sentences and include that as the first user/system message for the next agent.
1. Reset or prune session state per agent
◦ Start each agent session with a fresh short history or trimmed context.
◦ If you must preserve earlier facts (e.g., name, account id), store those in metadata and re-inject only the necessary fields.
1. Reuse warmed agent instances when possible
◦ Keep a pool of warm agents to avoid cold-init overhead. Initialize tool lists but keep prompts minimal until assigned.
1. Measure per-agent usage
◦ Track tokens per agent separately (tag usage_collector with agent id) so you can see which agent contributes most tokens and optimize it.
Implementation sketch (LiveKit-like)
1. Create a factory for agents and sessions:
Copy code
def create_agent_session(agent_config, short_system_prompt):
model = create_model_for_agent(agent_config)
session = AgentSession(llm=model, vad=self.silero_vad, ...)
session.metadata = {...}
# minimal on-metrics wiring
session.on("metrics_collected", lambda ev: usage_collector.collect(ev.metrics, agent=agent_config.name))
return session Copy code
next_session = create_agent_session(next_agent_config, next_system_prompt)
# Seed minimal context:
seed_messages = [
{"role":"system","content": next_system_prompt},
{"role":"system","content": f"handoff: {json.dumps(required_facts)}"},
{"role":"user","content": summary},
]
await next_session.send_seed_messages(seed_messages)b
boundless-author-7811
03/04/2026, 9:10 AMa concrete code handoff example tailored to your existing _create_agent_and_session flow
t
tall-airline-16756
03/04/2026, 9:11 AMI don't have the answer you're looking for.
You could also try asking your question:
• in one of the other Slack channels
or
• to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code
If you find the answer, please post it here to help others!
b
boundless-author-7811
03/04/2026, 9:18 AMa concrete code handoff example tailored to your existing _create_agent_and_session flow
7 Views