https://livekit.io logo
Join Slack
Powered by
- llm_request: APITimeoutError: Request timed out....
# ask-ai
b
• llm_request: APITimeoutError: Request timed out. (body=None, retryable=True) • llm_request_run: APITimeoutError: Request timed out. (body=None, retryable=True) • llm_request: APITimeoutError: Request timed out. (body=None, retryable=True) • llm_request_run: APITimeoutError: Request timed out. (body=None, retryable=True) • llm_request: APITimeoutError: Request timed out. (body=None, retryable=True) • llm_request_run: APITimeoutError: Request timed out. (body=None, retryable=True) • llm_request: APITimeoutError: Request timed out. (body=None, retryable=True) • llm_request_run: APITimeoutError: Request timed out. (body=None, retryable=True) I am getting this ssieu and I have already teh fallbck LLM return AgentSession( # --------------------------------------------------------------------- # CORE MODELS (STT, LLM, TTS) # --------------------------------------------------------------------- # STT (Speech-to-Text): Converts audio to text. # Uses a fallback adapter: Tries your primary model first; if it fails, falls back to Cartesia. _stt_*=*FallbackSTTAdapter( _stt_*=*[ inference.STT(_model_*=*settings.stt_model), inference.STT(_model_*=*settings.fallback_stt_model), ] ), # LLM (Large Language Model): The "brain" generating responses. # Uses a fallback adapter: Tries OpenAI first; if it errors, falls back to Gemini Flash. # 'frequency_penalty' reduces repetition; 'presence_penalty' encourages new topics. _llm_*=*FallbackLLMAdapter( _llm_*=*[ inference.LLM( _model_*=*settings.llm_model, ), inference.LLM( _model_*=*settings.fallback_llm_model, ), ] ), # TTS (Text-to-Speech): Converts text responses back to audio. _tts_*=get_fallback_tts_providers(_tts_provider_=tts_provider, voice=*voice), _tools_*=*[end_call], # --------------------------------------------------------------------- # VAD (Voice Activity Detection) - "Is the user speaking?" # Using centralized config from constants.py # --------------------------------------------------------------------- _vad_*=*vad_model or silero.VAD.load( _min_speech_duration_*=*settings.vad_min_speech_duration, _min_silence_duration_*=*settings.vad_min_silence_duration, _activation_threshold_*=*settings.vad_activation_threshold, _prefix_padding_duration_*=*settings.vad_prefix_padding_duration, _max_buffered_speech_*=*settings.vad_max_buffered_speech, _sample_rate_*=*settings.vad_sample_rate ), _turn_detection_*=*MultilingualModel(), # Endpointing: The MAIN speed knob _min_endpointing_delay_*=*settings.min_endpointing_delay, _max_endpointing_delay_*=*settings.max_endpointing_delay, # Interruptions: Keep your safe values _min_interruption_duration_*=*settings.min_interruption_duration, _min_interruption_words_*=*settings.min_interruption_words, _false_interruption_timeout_*=*settings.false_interruption_timeout, _resume_false_interruption_*=*settings.resume_false_interruption, # Critical speed settings _preemptive_generation_*=*settings.preemptive_generation, _allow_interruptions_*=*settings.allow_interruptions, _discard_audio_if_uninterruptible_*=*settings.discard_audio_if_uninterruptible, _user_away_timeout_*=*settings.user_away_timeout, )
✅ 1
t
Thanks for reaching out, Yethu. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I can see the repeated 
APITimeoutError
entries and the log lines showing the LLM failing, several “recovery failed” attempts, and one eventual “recovered” message. I’ll help you troubleshoot why the fallback LLM isn’t preventing these failures and how to reduce the timeouts. Quick summary of what the logs show (what I’m responding to) • Repeated APITimeoutError on LLM requests and several “LLM recovery failed” warnings. • One later INFO that the fallback adapter recovered. • Related errors/warnings: “speech not done in time after interruption” and a note about preemptive generation with changed chat context/tools. • Also a deprecation warning about RoomInputOptions/RoomOutputOptions (not the cause of timeouts, but worth updating). What you can try next (prioritized) 1. Reduce LLM work per request ◦ Lower the 
max_tokens
(or equivalent) you send to the LLM so responses finish faster. ◦ Avoid very long system+history context during high-latency periods; send a smaller recent context. 1. Switch to streaming/incremental mode (if you aren’t already) ◦ Ensure the LLM calls use streaming responses (time-to-first-token matters). Streaming reduces the chance of hitting client-side timeouts waiting for full responses. 1. Tune timeouts and retries ◦ Increase client-side request timeout slightly so transient slowness doesn’t mark the provider as failed prematurely. ◦ Ensure the fallback adapter’s retry/backoff configuration allows enough time for the provider to recover between attempts. 1. Investigate network and instance resources ◦ Check network latency to the LLM provider’s API (packet loss/high RTT increases timeouts). ◦ If running on burstable cloud instances (e.g., AWS t3/t4), verify CPU credits and consider switching to sustained-performance instance types — CPU starvation can cause long inference delays. 1. Confirm fallback configuration and ordering ◦ Verify the 
FallbackLLMAdapter
is correctly configured with the intended primary and fallback models and that credentials for the fallback provider are valid. ◦ Confirm the fallback provider supports streaming and the same request parameters (token limits, param names). Some fallbacks behave differently and can be slower. 1. Address related runtime warnings ◦ The “speech not done in time after interruption” indicates speech generation (TTS) also hit timing constraints; reducing LLM latency will often help TTS finish on time. ◦ The preemptive generation warning means preemptive responses are being generated while chat context or tools change; ensure you only enable preemptive generation when context/tools are stable, or disable it until the timeout issues are resolved. Targeted diagnostics I’ll need from you to narrow this down • Which LLM providers/models are configured for primary and fallback? (e.g., OpenAI gpt-4, Gemini Flash) • Current client-side timeout and retry/backoff settings for LLM calls. • Whether you’re using streaming responses or waiting for full responses. • Are you seeing high network latency from your agent hosts to the LLM APIs? Any recent network changes? • Are the agent hosts on burstable cloud instances (e.g., t3/t4) or other low-powered machines? • A short time range of raw logs (including the timestamps around failures) beyond the screenshot, if available — especially any HTTP status codes or error payloads from the LLM provider. If you prefer immediate quick actions • Temporarily lower 
max_tokens
and enable streaming. • Increase client timeout by a small factor and add a slightly longer backoff between retries. • If using burstable instances, move to a non-burst instance or scale up CPU for a short test period. What I can do next for you • Walk through how to change timeouts, enable streaming, or update 
max_tokens
in your current LiveKit agent config (tell me which adapter code or SDK you’re using). • Review a paste of the relevant portion of your configuration and recent logs and suggest concrete config changes. Which of the diagnostics above can you provide now? That will let me give the exact configuration edits or commands to try next. Sources: Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS, with interactive elements
b
how can I check streaming is enabled or not
t
Thanks — I’ll help you check whether streaming is enabled for your LLM calls. I’m responding to your question about how to verify streaming so we can reduce LLM timeouts and let TTS start sooner. Quick checklist to confirm streaming is enabled • Check your AgentSession / LLM adapter config ◦ Open the code where you construct the LLM adapter (your snippet shows 
inference.LLM(model=...)
inside 
FallbackLLMAdapter
). Look for a streaming-related flag or parameter such as 
stream=True
, 
streaming=True
, 
use_stream=True
, or use of an `LLMStream`/`ChatChunk` interface. ◦ If you see 
stream
set to true (or an explicit streaming client/stream class used), streaming is enabled. If there’s no streaming flag or it’s false, streaming is not enabled. • Verify provider/model supports streaming ◦ Confirm the primary and fallback models you configured actually support streaming (e.g., many OpenAI and other modern providers have streaming endpoints). If a fallback model does not support streaming, the adapter may fall back to non-streaming behavior when using it. • Check runtime logs / metrics for time-to-first-token ◦ Look for metrics named 
llm_ttft
, 
time_to_first_token
, or log lines indicating tokens/chunks arriving incrementally. A low TTFT (e.g., <300ms) and log entries showing partial tokens/chunks indicate streaming is working. ◦ If logs only show a single completion or you only see “request completed” with no intermediate chunks, likely streaming is not active. • Look for use of streaming interfaces in code ◦ Search your codebase for 
LLMStream
, 
ChatChunk
, or handlers that process partial token events. If your agent processes 
ChatChunk
objects, streaming is in use. How to test quickly 1. Force a short prompt and watch logs: ◦ Send a prompt that should produce output slowly (e.g., “Explain step by step…”) and watch the agent logs. With streaming you should see incremental tokens/chunks or repeated events for partial output before final completion. 1. Measure TTFT: ◦ Send a request and record how long until first token arrives. If your client records 
llm_ttft
or similar metrics, check them. If time to first token is near the full completion time, streaming is not functioning. 1. Temporarily switch to a known-streaming model: ◦ If you can, set the primary LLM to a model you know supports streaming and run the test above to confirm your code can handle streaming responses. If streaming is not enabled — how to enable it • Add or set the streaming flag on your LLM adapter (example names: 
stream=True
, 
streaming=True
), or instantiate the streaming client provided by your LLM integration (e.g., an 
LLMStream
class). • Ensure your response-processing path handles partial outputs (e.g., handlers for 
ChatChunk
) and that TTS is able to start synthesis from partial text. • Keep 
preemptive_generation
enabled only if your pipeline is stable with streaming; it complements streaming by starting generation earlier. What I need from you to give exact steps • The exact adapter/class you use to create the LLM in your code (the file/class around 
inference.LLM(...)
). • Which LLM providers/models you have configured for primary and fallback (e.g., OpenAI gpt-4, Gemini Flash). • A short snippet of the LLM adapter configuration (if possible) — I can point to the exact parameter name and show the change. If you share the adapter snippet or model names, I’ll provide the exact code/setting to check or enable streaming in your setup. Sources: Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | Agent session | LiveKit Documentation | How to match the latency of the homepage agent | LiveKit | Agent speech and audio | LiveKit Documentation | LLM Metrics | LiveKit Documentation, with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by