https://livekit.io logo
Join Slack
Powered by
how to catch the errors and send an alert when no ...
# ask-ai
f
how to catch the errors and send an alert when no voice can be produced because of complete failure of STT, LLM, TTS or live livekit.agents ?
✅ 1
t
Thanks for reaching out, Dmitry. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about detecting failures and alerting when STT, LLM, TTS, or LiveKit agents produce no voice. High-level approach (what you can do) • Instrument each pipeline stage (STT → LLM → TTS → agent orchestration) to emit structured events for success, error, and timings. • Define an expected response window after user input (e.g., 800ms target, alert if >3s) and treat missing audio within that window as a failure condition. • Log errors centrally and send alerts from those logs (or from real‑time monitors) to your alerting system (PagerDuty, Slack, email, etc.). Concrete detection techniques • STT: detect empty transcripts, very low-confidence results, or transcription exceptions. Raise a ToolError from function tools so the LLM can handle it or log/alert immediately. • LLM: capture timeouts, API errors, and “no output” cases. Use explicit per-call timeouts and fallbacks (FallbackAdapter) and log model error spans. • TTS: catch synthesis exceptions and monitor TTFB/streaming audio events. If synthesis returns no bytes or streaming stalls, mark as failure. • LiveKit agents / SDK: watch agent worker lifecycle logs (draining/stuck), session-level observability (trace spans, logs), and room/webhook events (egress failures, abrupt disconnects). Quick implementation pattern 1. Emit structured metrics/events per turn: ◦ e2e_latency, stt_confidence, stt_text_length, llm_status, tts_bytes, tts_error 1. Use these rules to decide failure: ◦ If 
stt_text_length == 0
OR 
stt_confidence < threshold
→ flag STT failure. ◦ If 
llm_status == timeout || llm_output == empty
→ flag LLM failure. ◦ If 
tts_bytes == 0 || tts_error
→ flag TTS failure. ◦ If e2e_latency > alert_threshold (e.g., 3s) and no audio emitted → flag pipeline failure. 1. On flag → Log error, increment error counter, and trigger alerting policy (notify on single critical failures or after N occurrences within T window). Best practices to avoid silent failures • Use ToolError in function tools to propagate STT/TTS/other tool failures into agent logic so the LLM can ask the user to repeat instead of producing silence. • Add fallback providers and timeouts for external LLM/TTS services. • Implement server-side retries for transient network issues, but avoid long blocking retries that delay user feedback. • Stream logs and traces to an observability platform (Datadog, CloudWatch, Sentry) via log drains so you can correlate session traces, spans, and errors. Alerting sources to wire up • Real-time per-turn metrics and alerts (latency, empty audio) • Session/agent trace errors (Agent Observability spans & logs) • Server/process-level logs via log drains for crashes/startup failures • Webhook lifecycle events (egress_ended errors, abrupt participant disconnects) If you want, I can: • Provide a short code snippet for raising 
ToolError
on empty STT results. • Draft the exact metric names and alert thresholds you should emit. Which of those would you like next, and which component (STT, LLM, TTS, or agent) should I show the snippet for? Sources: Sequential Pipeline Architecture for Voice Agents | LiveKit | Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit | Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit | Function tool definition | LiveKit Documentation | Agent deployment overview | LiveKit Documentation, with interactive elements
f
yes provide pls
t
Thanks — I’ll show a concise example that (1) detects empty STT results, (2) raises 
ToolError
so the agent can handle it, and (3) logs + triggers an alert. I’ll use Python-style pseudocode that matches LiveKit function tools. Code snippet — STT tool that raises ToolError on empty transcription
Copy code
from livekit.agents.llm import ToolError
import logging
import requests  # or your async http client

logger = logging.getLogger("voice_agent")

@function_tool()
async def transcribe_audio(self, context: RunContext, audio_bytes: bytes) -> str:
    try:
        # call your STT provider (sync/async depending on client)
        resp = await stt_client.transcribe(audio_bytes, timeout=5.0)
    except Exception as exc:
        logger.exception("STT service call failed")
        trigger_alert("STT service error", {"stage": "stt", "error": str(exc)})
        raise ToolError(f"STT service failed: {exc}")

    # Inspect result for empty output or low confidence
    text = getattr(resp, "text", "") or ""
    confidence = getattr(resp, "confidence", None)

    if not text.strip() or (confidence is not None and confidence < 0.3):
        logger.warning("Empty/low-confidence transcription", extra={"confidence": confidence})
        trigger_alert("Empty or low-confidence STT result", {"stage": "stt", "confidence": confidence})
        # Raise ToolError so the LLM receives a structured error and can ask user to repeat
        raise ToolError("STT returned empty or low-confidence transcription. Please retry.")

    return text
Logging + alert helper (example)
Copy code
def trigger_alert(message: str, details: dict = None):
    # Send to your alerting system (PagerDuty, Slack webhook, Sentry, etc.)
    payload = {"message": message, "details": details or {}}
    # example: post to internal alert webhook
    try:
        requests.post("https://alerts.example.com/webhook", json=payload, timeout=2.0)
    except Exception:
        logger.exception("Failed to send alert")
Minimal TTS & LLM guards (patterns) • Wrap calls in try/except, log exceptions and call 
trigger_alert(...)
. • If TTS returns no bytes or streaming stalls, log + alert and raise ToolError from the TTS tool. • For LLM call, set a timeout; on timeout or empty output, log + alert and return a structured error via 
ToolError
so flow can recover. Failure-detection rule you should emit per turn (metric names you can use) • 
stt_text_length
(int) • 
stt_confidence
(float) • 
llm_status
(enum: ok/timeout/error) • 
tts_bytes
(int) • 
e2e_latency_ms
(int) Simple alert rule examples • If 
stt_text_length == 0
-> immediate alert (or escalate after N occurrences) • If 
llm_status == timeout
-> alert immediately • If 
tts_bytes == 0
OR no audio produced within 3s of LLM output -> alert Next step Would you like a sample TTS tool snippet that raises ToolError on empty audio or an example alerting rule configuration (e.g., Sentry/Datadog) next? , with interactive elements
Open in Slack
PreviousNext
Powered by