how will i track the realtime metrics as metrics_collected event is deprecated

Thanks for reaching out, Harshita. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

how can i track realtime model metrics using per plugin metrics collected event? if session level metrics collected event is deprecated

Thanks — I’m responding about tracking realtime model metrics with the per-plugin

metrics_collected

approach now that session-level

metrics_collected

is deprecated. A few quick clarifying questions so I can give precise steps: • Which SDK and version are you using (web, iOS, Android, or server/agent SDK)? • Which realtime model metrics do you need (e.g., TTFT, duration, input/output tokens, tokens/sec, per-modality breakdown)? • Are you using any plugins already (STT, TTS, LLM plugin names) or a realtime model that emits

RealtimeModelMetrics

? What I can say right now: • Per-plugin metrics listeners are the recommended way to collect detailed metrics for individual pipeline stages (STT, LLM, TTS). • For realtime models,

RealtimeModelMetrics

are currently emitted by the session via `metrics_collected`; there is no documented per-plugin replacement for those realtime-model-specific metrics yet. • Options we can pursue once I know your SDK/version and goals: ◦ Show how to attach per-plugin

metrics_collected

listeners (for STT/LLM/TTS plugins) to capture per-stage latency and token stats. ◦ Show how to continue receiving

RealtimeModelMetrics

from the session (if your realtime model still emits them) and filter/forward them to your observability backend. ◦ Recommend using the Agent Console or exporting plugin-level metrics to your telemetry backend for real-time dashboards. Tell me the SDK/version and which specific metrics/plugins matter most and I’ll provide exact code snippets and steps. Sources: Understand and Improve Agent Latency | LiveKit | Agent Console | LiveKit Documentation | Debug voice agents in real time with Agent Console | LiveKit, with interactive elements

how to calculate the total latency per turn eou + llmttft+ttsttfb using on_conversation_item added? because here in this new metrics speech id is not there

Thanks — I’m responding about calculating total per‑turn latency (EOU + LLM TTFT + TTS TTFB) using

on_conversation_item_added

when

speech_id

is missing. Short answer: • Use the metrics attached to the assistant conversation item (

ev.item.metrics

). Prefer

e2e_latency

if present — it already represents the total end‑to‑end latency. If

e2e_latency

is not available, sum

llm_node_ttft

+

tts_node_ttfb

and add an estimate for EOU detection delay (if you have it). Quick checklist and example code (Python): 1. In

on_conversation_item_added

, read

m = ev.item.metrics

. 2. If

m.get("e2e_latency")

exists, use that as total latency. 3. Otherwise compute: ◦

llm = m.get("llm_node_ttft", 0)

◦

tts = m.get("tts_node_ttfb", 0)

◦

eou = m.get("end_of_turn_delay", estimated_value_if_missing)

◦

total = eou + llm + tts

Example:

@session.on("conversation_item_added")
def on_conversation_item_added(ev):
    if ev.item.role != "assistant":
        return
    m = ev.item.metrics or {}
    total = m.get("e2e_latency")
    if total is None:
        llm = m.get("llm_node_ttft", 0.0)
        tts = m.get("tts_node_ttfb", 0.0)
        eou = m.get("end_of_turn_delay", 0.0)  # replace 0.0 with your estimated EOU if you track it separately
        total = eou + llm + tts
    print(f"total_latency_per_turn={total:.3f}s")

Notes and recommendations: •

speech_id

is not required for this calculation because

e2e_latency

or the per-item metrics provide the per-turn numbers. • If

end_of_turn_delay

is not exposed, you can estimate EOU by measuring time between the last user audio event (or user item timestamp) and when the assistant item appears. • If you want stronger attribution across stages, attach per-plugin

metrics_collected

listeners for STT/LLM/TTS and correlate by timestamps (plugin metrics include timestamps even if

speech_id

is absent). If you share a sample

ev.item.metrics

payload from your SDK/version, I’ll give a precise snippet that matches the exact keys you receive. Sources: Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit | Understand and Improve Agent Latency | LiveKit, with interactive elements

status_code=400, retryable=True, request_id=b74f8155-36df-4e9b-81c0-325d76f3e010, body={'message': \"Unsupported parameter: 'messages'. In the Responses API, this parameter has moved to 'input'. Try again with the new parameter. See the API documentation for more information: https://platform.openai.com/docs/api-reference/responses/create.\", 'type': 'invalid_request_error', 'param': None, 'code': 'unsupported_parameter'}", "pid": 58070, "job_id": "AJ_VRC6J3YAumqN", "room_id": "RM_LD7Ybe9QKVcH", "timestamp": "2026-05-11T160613.924685+00:00"}

Thanks — I’m responding about the 400 error and how to fix the OpenAI Responses API parameter mismatch. What happened: • The Responses API no longer accepts `messages`; it requires the request body to use

input

. That caused the error: Unsupported parameter: 'messages'. How to fix (short): • Replace

messages

with

input

in the request you send to the Responses endpoint or use LiveKit’s

openai.responses.LLM()

which uses the Responses API shape. If you want, tell me which code path you’re using and I’ll provide an exact patch: • Are you calling OpenAI directly or using LiveKit’s OpenAI plugin? • Share the SDK/language (Python, Node, etc.) and a small snippet of your request. Sources: Azure OpenAI Realtime API and LiveKit | LiveKit Documentation | OpenAI LLM models | LiveKit Documentation | Debug voice agents in real time with Agent Console | LiveKit | Agent Console | LiveKit Documentation, with interactive elements

using openai_withazure

Thanks — you're using

openai_withazure

. The 400 is because the OpenAI Responses API expects

input

, not

messages

. For Azure Reponses usage with LiveKit: • Use the Responses-style payload: replace any

messages

field with

input

. • In LiveKit, prefer

openai.responses.LLM()

/

RealtimeModel.with_azure()

patterns so the SDK sends the correct shape for you. Quick checklist to fix: 1. If calling OpenAI directly, change: ◦ From: {"messages": [...]} ◦ To: {"input": [...]} (keep the same message objects inside

input

per Responses API) 1. If using LiveKit plugin, ensure you instantiate the Azure provider as the Responses variant (example:

RealtimeModel.with_azure(...)

or

openai.responses.LLM(...)

) rather than the older Chat/`messages` style. 2. Confirm your Azure deployment/endpoint and that you’re using the Reponses-compatible endpoint (wss/HTTP as required). If you share your SDK/language and the request snippet you’re sending (or your LiveKit plugin config), I’ll give an exact code change. , with interactive elements

how to use azure gpt-5-mini in agent session then?

Thanks — I’ll show how to run Azure

gpt-5-mini

in an AgentSession and give two options (LiveKit-managed inference or direct Azure plugin). Which SDK are you using: Python or Node? Quick guidance: • Option A — LiveKit Inference (easiest, no Azure keys): ◦ Set the LLM to

openai/gpt-5-mini

and

provider="azure"

in the

inference.LLM

config. ◦ Python:

from livekit.agents import AgentSession, inference

    session = AgentSession(
        llm=inference.LLM(
            model="openai/gpt-5-mini",
            provider="azure",
            extra_kwargs={"temperature": 0.7, "max_completion_tokens": 512}
        ),
        # tts/stt/vad...
    )

• Node:

import { AgentSession, inference } from '@livekit/agents';

    const session = new AgentSession({
      llm: new inference.LLM({
        model: "openai/gpt-5-mini",
        provider: "azure",
        modelOptions: { temperature: 0.7, max_completion_tokens: 512 }
      }),
      // tts/stt/vad...
    });

• Option B — Azure OpenAI plugin (you manage Azure keys/deployment): ◦ Deploy

gpt-5-mini

in your Azure resource, then configure the plugin to use your Azure endpoint, deployment name, and key. ◦ Python (plugin):

from livekit.plugins import openai

    session = AgentSession(
        llm=openai.LLM.with_azure(
            azure_deployment="<your-deployment-name>",
            azure_endpoint="https://<your-endpoint>.openai.azure.com",
            api_key="<your-api-key>",
            api_version="2024-10-01-preview",
        ),
        # tts/stt/vad...
    )

• Node (plugin): use the analogous

withAzure

options and supply

azureDeployment

,

azureEndpoint

, and

apiKey

. Notes: • Model ID to use:

openai/gpt-5-mini

. • Choose LiveKit Inference for simplicity (no Azure creds). Choose the Azure plugin if you need to control billing, rate limits, or use your own Azure deployment. • If you want exact code for your project, send your SDK (Python/Node), and whether you prefer LiveKit Inference or direct Azure, and I’ll provide the exact snippet and any env vars to set. Sources: OpenAI LLM models | LiveKit Documentation | Azure OpenAI LLM plugin guide | LiveKit Documentation | Build voice, video, and physical AI | LiveKit, with interactive elements