https://livekit.io logo
Join Slack
Powered by
give all the docs and details needed to Leverage t...
# ask-ai
s
give all the docs and details needed to Leverage the Agents metrics API to track and optimize latency at each stage (VAD, STT, LLM, TTS). Key metrics include Time To First Token (TTFT) and Time To First Byte (TTFB).
t
Thanks for reaching out, Rhythm. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
The Agents metrics API enables detailed tracking and optimization of latency across all stages of the voice agent pipeline: VAD (Voice Activity Detection), STT (Speech-to-Text), LLM (Large Language Model), and TTS (Text-to-Speech). Key Latency Metrics • Time To First Token (TTFT): Measures how long it takes for the LLM to generate the first token of its response. • Time To First Byte (TTFB): Measures how long it takes for the TTS model to generate the first byte of audio output. Metrics by Stage • VAD: 
idle_time
, 
inference_duration_total
, 
inference_count
• STT: 
transcription_delay
, 
audio_duration
, 
streamed
• LLM: 
ttft
, 
input_tokens
, 
output_tokens
, 
tokens_per_second
• TTS: 
ttfb
, 
audio_duration
, 
characters_count
• End-of-Utterance (EOU): 
end_of_utterance_delay
Calculating Total Conversation Latency
Copy code
total_latency = eou.end_of_utterance_delay + llm.ttft + tts.ttfb
Implementation Example (Python)
Copy code
from livekit.agents.metrics import LLMMetrics, STTMetrics, TTSMetrics, EOUMetrics, VADMetrics

class CombinedMetricsAgent(Agent):
    def __init__(self):
        llm = openai.LLM(model="gpt-4o-mini")
        stt = deepgram.STT()
        tts = openai.TTS()
        silero_vad = silero.VAD.load()
        llm.on("metrics_collected", self.on_llm_metrics_collected)
        stt.on("metrics_collected", self.on_stt_metrics_collected)
        stt.on("eou_metrics_collected", self.on_eou_metrics_collected)
        tts.on("metrics_collected", self.on_tts_metrics_collected)
        silero_vad.on("metrics_collected", self.on_vad_event)
Optimization Strategies • Deploy agents close to LLM, STT, and TTS services to minimize network latency • Monitor TTFT and TTFB to identify and address bottlenecks • Select low-latency providers for each pipeline component • Fine-tune configuration parameters for each stage The 
RealtimeModelMetrics
class and the provided SDK metric classes allow you to systematically monitor and optimize each stage of your voice agent's performance.
s
getting this error: # Create OpenAI Realtime model with multimodal capabilities model = openai.realtime.RealtimeModel( voice="shimmer", temperature=0.8, model="gpt-4o-realtime-preview-2025-06-03", #model="gpt-4o-mini-realtime-preview", # Tell Whisper to expect Turkish (“tr”) audio _input_audio_transcription_=InputAudioTranscription( # model="gpt-4o-transcribe", language="tr" ), _turn_detection_=None, modalities=["text"] ) # Create AgentSession with the multimodal model session = AgentSession( llm=model, tts=elevenlabs.TTS( language=LANG_MODE, _voice_id_="cgSgspJ2msm6clMCkdW9", _streaming_latency_=3, ), NG livekit.agents: failed to publish transcription Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 237, in _publish_transcription await self._room.local_participant.publish_transcription(transcription) File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/participant.py", line 285, in publish_transcription raise PublishTranscriptionError(cb.publish_transcription.error) livekit.rtc.participant.PublishTranscriptionError: failed to send transcription, room closed: channel closed 2025-07-21 161858,321 - WARNING livekit.agents - failed to publish transcription Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 237, in _publish_transcription await self._room.local_participant.publish_transcription(transcription) File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/participant.py", line 285, in publish_transcription raise PublishTranscriptionError(cb.publish_transcription.error) livekit.rtc.participant.PublishTranscriptionError: failed to send transcription, room closed: channel closed {"pid": 8698, "job_id": "AJ_AsCimJYAWruu"} [2025-07-21 161858,323] INFO __mp_main__: 🔄 Finalizing call - starting information extraction... 2025-07-21 161858,323 - INFO __mp_main__ - 🔄 Finalizing call - starting information extraction... {"pid": 8698, "job_id": "AJ_AsCimJYAWruu"} [2025-07-21 161858,323] INFO __mp_main__: 🧠 Starting LLM-based information extraction from conversation... 2025-07-21 161858,323 - INFO __mp_main__ - 🧠 Starting LLM-based information extraction from conversation... {"pid": 8698, "job_id": "AJ_AsCimJYAWruu"} [2025-07-21 161858,323] WARNING __mp_main__: ⚠️ No conversation transcript available for extraction 2025-07-21 161858,323 - WARNING __mp_main__ - ⚠️ No conversation transcript available for extraction {"pid": 8698, "job_id": "AJ_AsCimJYAWruu"} [2025-07-21 161858,323] WARNING __mp_main__: ⚠️ No information extracted by LLM. Call data will be incomplete. 2025-07-21 161858,323 - WARNING __mp_main__ - ⚠️ No information extracted by LLM. Call data will be incomplete. {"pid": 8698, "job_id": "AJ_AsCimJYAWruu"} [2025-07-21 161858,323] WARNING livekit.agents: failed to publish transcription Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 361, in capture_text self._writer = await self._create_text_writer() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 337, in _create_text_writer return await self._room.local_participant.stream_text( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/participant.py", line 578, in stream_text await writer._send_header() File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/data_stream.py", line 189, in _send_header raise ConnectionError(cb.send_stream_header.error) ConnectionError: engine: connection error: engine is closed [2025-07-21 161858,323] WARNING livekit.agents: failed to publish transcription Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 361, in capture_text self._writer = await self._create_text_writer() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 337, in _create_text_writer return await self._room.local_participant.stream_text( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/participant.py", line 578, in stream_text await writer._send_header() File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/data_stream.py", line 189, in _send_header raise ConnectionError(cb.send_stream_header.error) ConnectionError: engine: connection error: engine is closed 2025-07-21 161858,323 - WARNING livekit.agents - failed to publish transcription Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 361, in capture_text self._writer = await self._create_text_writer() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 337, in _create_text_writer return await self._room.local_participant.stream_text( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/participant.py", line 578, in stream_text await writer._send_header() File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/data_stream.py", line 189, in _send_header raise ConnectionError(cb.send_stream_header.error) ConnectionError: engine: connection error: engine is closed {"pid": 8698, "job_id": "AJ_AsCimJYAWruu"} [2025-07-21 161858,324] WARNING livekit.agents: failed to publish transcription Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 237, in _publish_transcription await self._room.local_participant.publish_transcription(transcription) File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/participant.py", line 285, in publish_transcription raise PublishTranscriptionError(cb.publish_transcription.error) livekit.rtc.participant.PublishTranscriptionError: failed to send transcription, room closed: channel closed [2025-07-21 161858,324] WARNING livekit.agents: failed to publish transcription Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 237, in _publish_transcription await self._room.local_participant.publish_transcription(transcription) File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/participant.py", line 285, in publish_transcription raise PublishTranscriptionError(cb.publish_transcription.error) livekit.rtc.participant.PublishTranscriptionError: failed to send transcription, room closed: channel closed 2025-07-21 161858,324 - WARNING livekit.agents - failed to publish transcription Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/voice/room_io/_output.py", line 237, in _publish_transcription await self._room.local_participant.publish_transcription(transcription) File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/rtc/participant.py", line 285, in publish_transcription raise PublishTranscriptionError(cb.publish_transcription.error) livekit.rtc.participant.PublishTranscriptionError: failed to send transcription, room closed: channel closed {"pid": 8698, "job_id": "AJ_AsCimJYAWruu"} [2025-07-21 161858,389] ERROR livekit.plugins.openai: failed to handle event Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/plugins/openai/realtime/realtime_model.py", line 759, in _recv_task self._handle_response_audio_delta( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ ResponseAudioDeltaEvent.construct(**event) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/plugins/openai/realtime/realtime_model.py", line 1341, in _handle_response_audio_delta item_generation.audio_ch.send_nowait( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ rtc.AudioFrame( ^^^^^^^^^^^^^^^ ...<4 lines>... ) ^ ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/utils/aio/channel.py", line 95, in send_nowait raise ChanClosed livekit.agents.utils.aio.channel.ChanClosed [2025-07-21 161858,389] ERROR livekit.plugins.openai: failed to handle event Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/plugins/openai/realtime/realtime_model.py", line 759, in _recv_task self._handle_response_audio_delta( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ ResponseAudioDeltaEvent.construct(**event) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/plugins/openai/realtime/realtime_model.py", line 1341, in _handle_response_audio_delta item_generation.audio_ch.send_nowait( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ rtc.AudioFrame( ^^^^^^^^^^^^^^^ ...<4 lines>... ) ^ ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/utils/aio/channel.py", line 95, in send_nowait raise ChanClosed livekit.agents.utils.aio.channel.ChanClosed 2025-07-21 161858,389 - ERROR livekit.plugins.openai - failed to handle event Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/plugins/openai/realtime/realtime_model.py", line 759, in _recv_task self._handle_response_audio_delta( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ ResponseAudioDeltaEvent.construct(**event) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/plugins/openai/realtime/realtime_model.py", line 1341, in _handle_response_audio_delta item_generation.audio_ch.send_nowait( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ rtc.AudioFrame( ^^^^^^^^^^^^^^^ ...<4 lines>... ) ^ ) ^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/livekit/agents/utils/aio/channel.py", line 95, in send_nowait raise ChanClosed livekit.agents.utils.aio.channel.ChanClosed {
t
Hey there! I noticed you're running into some tricky connection errors that can be super frustrating. Let me break down what's happening when you see those "failed to publish transcription" or "engine connection error" messages. What's Going On? • Your connection might have dropped unexpectedly (like if everyone left the room or there was a network hiccup) • The RTC engine could have shut down or lost its connection • You're trying to publish or stream after the connection is already gone Don't Worry, Here Are Some Friendly Tips to Fix This 1. Double-Check Your Connection Before doing anything, make sure you're still connected:
Copy code
if self._room.isconnected():
    await self._room.local_participant.publish_transcription(transcription)
2. Stay Prepared for Reconnections Keep an ear out for when things might reconnect:
Copy code
self._room.on("reconnected", self._on_reconnected)

def _on_reconnected(self):
    # Republish tracks or reinitialize as needed
    ...
3. Check the Engine's Status If the engine is closed, just take a pause:
Copy code
if self._room.engine.isClosed:
    return
4. Build in Some Error Protection Add some safety nets to catch potential issues:
Copy code
try:
    if self._room.isconnected():
        await self._room.local_participant.publish_transcription(transcription)
except Exception as e:
    logger.warning("failed to publish transcription", exc_info=e)
    # Optionally implement retry or fallback
5. Keep an Eye Out for Room Closures Watch for signals like 
ROOM_CLOSED
, 
ROOM_DELETED
, or 
SIGNAL_CLOSE
to manage things smoothly. 6. Exit Gracefully Always disconnect rooms nicely using 
Room.disconnect()
to avoid any abrupt goodbyes. I know connection issues can be a real pain, but with these strategies, you'll be handling those tricky errors like a pro! These tips should help you stay connected and keep things running smoothly. You've got this! 😊 Sources: livekit-android-sdk/src/main/java/io/livekit/android/room/RTCEngine.kt | src/room/RTCEngine.ts | Connecting to LiveKit | LiveKit Docs | livekit/src/rtc_engine/rtc_session.rs | livekit-agents/livekit/agents/voice/room_io/_output.py
2 Views
Open in Slack
PreviousNext
Powered by