what is duration in LLM metrics?

Thanks for reaching out, Rashmi. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Is it same as the latency?

Hey there! Just wanted to clarify the difference between "Duration" and "Latency" in LiveKit—they’re related but not quite the same: • Duration: This is the total time it takes for the LLM to generate the whole completion, from start to finish. • Latency: This specifically means the Time to First Token (TTFT), or how long it takes for the LLM to produce the very first token of the response. So, to sum it up, latency (TTFT) is actually part of the overall duration, but they each highlight different parts of how the LLM performs. Sources: Capturing metrics | LiveKit Docs | livekit-agents/livekit/agents/llm/llm.py

What does Time to First Token mean?

Time to First Token (TTFT) is all about how quickly the Language Model (LLM) starts replying—it measures the time in seconds it takes to generate that very first token after it begins processing your request. This is super important, especially if you need responses in real-time and want things to feel snappy. So, if the TTFT is 0.5 seconds, that means the LLM kicks off its response in just half a second. Keeping an eye on TTFT can really help spot any slowdowns and make sure everything runs as smoothly and quickly as possible. Sources: Agent Testing & Evaluation | livekit/livekit_composite | DeepWiki

okay. So, how can I expose latency?

Here’s a friendly guide to help you expose latency metrics in LiveKit: 1. Get to Know the Predefined Metrics: LiveKit already gives you handy latency metrics like

publisherRtt

,

subscriberRtt

, and

ttfb

(Time To First Byte) to keep an eye on. 2. Use the MetricsCollector: You can use the

MetricsCollector

class to track latency metrics easily. For example:

from livekit.agents.metrics import MetricsCollector

metrics = MetricsCollector()
metrics.record_latency("llm_response_time", response_time_ms)

3. Turn On Telemetry: Make sure telemetry is enabled in your LiveKit setup so all those useful metrics get captured and sent your way. 4. Check Out Metrics with Prometheus: You can query latency metrics using Prometheus like this:

prometheus_record_latency_seconds_bucket{label="subscriberRtt"}

5. Dig Into the Data: Use what you’ve collected to monitor how things are running and find ways to make your app even better. Following these steps will make it a breeze to expose and keep track of latency metrics in your LiveKit app. If you have any questions, I’m here to help! Sources: Sources/LiveKit/Protos/livekit_metrics.pb.swift | livekit/livekit_internal.pb.go | Agent Testing & Evaluation | livekit/livekit_composite | DeepWiki | pkg/sfu/forwardstats.go

what is publisherRtt and subscriberRtt?

publisherRtt and subscriberRtt are handy little metrics in LiveKit that help us understand network latency: • publisherRtt: This tells us the round-trip time (RTT) from a participant all the way to the server. • subscriberRtt: This one measures the RTT from the server back to the participant. We use these metrics to keep an eye on connection quality and how the network is performing, so we can make sure your media streams smoothly and adjust things as network conditions change. It’s all about giving you the best experience possible! Sources: Protocol & WebRTC | livekit/livekit_composite | DeepWiki | Sources/LiveKit/Protos/livekit_rtc.pb.swift

what is ttft

TTFT stands for Time to First Token. It basically tells us how long, in seconds, it takes for the Language Model (LLM) to come up with the very first token of a response once it starts working on your request. This is a super important measure because it helps us understand how quickly the system can respond—especially when you need answers in real-time and speed really matters.

can we dump the AgentMetrics on the logs?