Thanks — I’ll help you design catching and handling errors for STT, LLM, and TTS in your pipeline (examples: balance exhausted, service unavailable, rate limits). Acknowledging scope: you want per-stage error handling for a sequential pipeline (VAD → STT → LLM → TTS). Below are concrete patterns and short examples you can apply at each stage. 1) General principles • Treat each stage independently: detect, categorize, handle, and report. • Classify errors: transient (rate limit, timeout), permanent (balance exhausted, auth failure), and data-quality (low STT confidence, empty transcript). • Use retries with backoff only for transient errors; do not retry for permanent errors. • Surface user-friendly messages or fallbacks (short TTS clip, “I’m having trouble right now”) instead of silence. 2) Stage-specific guidance • STT ◦ Detect: check empty transcript or low confidence score. ◦ Handle: ask user to repeat or switch to a simpler STT provider. ◦ Example actions: ▪︎ If confidence < threshold → prompt: “Sorry, I didn’t catch that — could you repeat?” ▪︎ If service unavailable or rate-limited → attempt failover STT provider or short exponential retry (max 2 tries), then fallback to text input option. • LLM ◦ Detect: timeouts,

ToolError

from downstream tool calls, rate limit responses, or specific API error codes. ◦ Handle: for timeouts/rate limits, retry once with backoff and shorter context; for permanent errors (balance/auth) decline generation and inform user. ◦ Example actions: ▪︎ On timeout → return quick fallback reply: “I’m still thinking — can I follow up?” and queue retry. ▪︎ On

ToolError

(external call failed) → let the agent reason about the failure and return a graceful message explaining the missing capability. ▪︎ If balance exhausted → stop invoking and inform admin/owner; present degraded mode to user. • TTS ◦ Detect: TTS API errors, empty audio, or playback failures. ◦ Handle: fallback to alternate TTS provider, fallback to prerecorded audio, or show text on UI. ◦ Example actions: ▪︎ On rate limit/unavailable → switch provider or play prerecorded message: “I can’t speak right now — here’s the text.” ▪︎ On repeated failure → disable voice output and continue via text. 3) Retry and backoff strategy (apply per-stage) 1. For transient errors: retry up to N times (e.g., 2) with exponential backoff (e.g., 500ms → 1500ms). 2. For throttling: honor

Retry-After

when provided. 3. For permanent errors: do not retry; escalate to admin/owner-facing notification. 4) Observability and isolation • Monitor each stage separately (logs, metrics, traces) so you can apply stage-specific retries and avoid cascading failures. • Emit structured errors (type, stage, code, message) so agent logic can decide: retry, fallback, or inform user. 5) How agents should surface failures to users • Be transparent but brief: explain what failed and the consequence. • Offer an action: retry, switch to text, try again later. • Example user message patterns: ◦ transient: “I’m having trouble reaching the speech service; trying again.” ◦ permanent: “We can’t use voice right now because the service balance is exhausted. Would you like to continue by text?” 6) Implementation notes (short) • Use

ToolError

for tool call failures so the LLM can reason about and communicate errors. • Enforce LLM timeouts and return a fallback if exceeded. • Implement provider failover lists and a simple health-check cache to avoid calling known-down providers. If you want, I can: • Provide a small code snippet for one stage (STT, LLM, or TTS) showing structured error types, retry/backoff, and fallbacks. • Or suggest exact thresholds (confidence, retries, timeouts) tailored to your latency/UX needs. Which stage would you like a code example for first? Sources: Sequential Pipeline Architecture for Voice Agents | LiveKit | The ReAct Pattern for Voice Agents and How AI Agents Think, Act, and Respon... | Function tool definition | LiveKit Documentation, with interactive elements