Thanks — I hear you: you’ve got a LiveKit server on EKS with Prometheus scraping 30+ metrics and it’s overwhelming to know what each metric means and which ones to watch. Below I’ll point you to the most relevant resources and give a short, practical list of the metric groups and the specific metrics I recommend you monitor first. Useful documentation and resources • LiveKit Prometheus and server observability overview: https://deepwiki.com/livekit/livekit_composite/1.4-server-side • Server SDK (Go) notes about telemetry & collected stats: https://deepwiki.com/livekit/livekit_composite/1.4.2-server-sdk-go • Agent observability & debugging dashboard (helps with latency/traces): https://livekit.com/products/agent-observability • Guide on latency for voice agents (explains key latency metrics and targets): https://livekit.com/field-guides/guide/reducing-latency-voice-agents-stt-tts-llm • How to create and publish custom metrics if you want to add focused telemetry: https://kb.livekit.io/articles/5701026044-creating-and-publishing-custom-metrics • Realtime model / pipeline metric examples: https://docs.livekit.io/recipes/metrics_realtime • Practical blog on understanding & improving agent latency: https://livekit.com/blog/understand-and-improve-agent-latency High-level metric groups and what they mean • SFU / server-level metrics ◦ Tracks SFU CPU, memory, active rooms, active participants, egress/ingress bytes. Useful to detect capacity issues and autoscaling signals. • Participant/session metrics ◦ Session duration, active connections per room, join/leave counts. Useful for usage and billing-related analysis. • Transport / WebRTC metrics (per track/participant) ◦ Bitrate (send/recv), packet loss, jitter, RTT. These show media quality and network-health problems. • Track-level metrics ◦ Frame rate, resolution, simulcast layer switches. Useful for video quality and if simulcast/adaptive bitrate is working. • Data channel / application-level metrics ◦ Bytes sent/received on data channels, message counts. Useful if you use data channels heavily (chat, control signals). • Telemetry / pipeline metrics (agents) ◦ Time-to-first-token (LLM TTFT), time-to-first-byte (TTS/TTFB), end-of-utterance delay. Critical for voice-agent latency tuning. • Custom metrics ◦ Any app-specific counters or histograms you add (API response times, model call durations). Which metrics to prioritize (quick checklist) 1. Server health and capacity ◦ CPU usage, memory usage, active_rooms, active_participants ◦ Why: tells you when to scale or investigate host resource limits. 1. Media quality (per room / per participant) ◦ bitrate_send/recv, packet_loss, jitter, rtt ◦ Why: direct indicators of call quality; packet loss and high RTT often explain user complaints. 1. Track-level quality for video ◦ fps, resolution, simulcast layer switches ◦ Why: helps identify encoding issues and whether adaptive bitrate is functioning. 1. Session and usage metrics ◦ join_count, leave_count, session_duration ◦ Why: usage trends and abnormal session churn detection. 1. Latency metrics for voice agents ◦ eou_delay, llm_ttft, tts_ttfb (or equivalent pipeline timings) ◦ Why: these explain end-to-end responsiveness — the single most important for voice UX. 1. Data channel traffic (if applicable) ◦ datachannel_bytes_sent/received ◦ Why: detect if data channel usage is saturating bandwidth or causing load. Practical steps to tame the metric noise • Start with the checklist above: add these as dashboards/alerts first. • Group metrics by category in dashboards (Server / Media / Session / Agent / Data) to reduce cognitive load. • Create alert rules for a small set of actionable signals only (e.g., packet_loss > 3% over 5m, p95 RTT > 300ms, CPU > 80%). • Use label filtering: focus on top N rooms or top N participants when investigating a problem instead of all metrics at once. • Export or aggregate metrics you don’t need raw (e.g., roll up per-room metrics to per-cluster summaries). If you want, I can help with next steps • I can map the specific metric names you’re seeing to the categories above (paste a short list of the metric names you have and I’ll annotate them). • I can propose a minimal Prometheus alerting rule set (3–6 alerts) and a simple Grafana dashboard layout for quick operational coverage. • I can help identify which metrics to instrument as custom metrics if your use case needs them. Which would you like me to do next — annotate your metric list, propose alerts, or build a dashboard layout? Sources: Server-side | livekit/livekit_composite | DeepWiki | server-sdk-go | livekit/livekit_composite | DeepWiki | Announcing LiveKit Cloud | LiveKit | Creating and Publishing Custom Metrics | How can I reduce latency in voice agents using STT, TTS and LLM? | LiveKit, with interactive elements