Thanks — I’m responding to your question about how shrinking the LLM prompt from 15,000 tokens to 1,000 tokens affects Time-to-First-Audio (TTFA) when using OpenAI for the

llm_node

and ElevenLabs for TTS. Short answer • Reducing the prompt from 15k → 1k tokens will mainly reduce the LLM’s Time-to-First-Token (TTFT), which directly reduces TTFA. • A realistic estimate is a TTFA improvement on the order of ~300–800 ms. The exact number depends on model choice and network/CPU conditions. • ElevenLabs TTS latency (Time-to-First-Byte / TTFB) stays essentially the same, so the LLM improvement is what moves the TTFA needle. Why this helps (concise) • TTFA ≈ TTFT (LLM) + TTFB (TTS). Shorter prompts reduce the LLM’s input-processing and serialization work, allowing the model to begin generation sooner. • OpenAI models’ TTFT is sensitive to input length; very large contexts (15k tokens) can add hundreds of milliseconds to over a second of delay before the first token is produced. • TTS (ElevenLabs) still must synthesize audio from the text; its startup latency doesn’t change when you shorten the prompt. Estimated numbers (example) • Before (15k tokens): LLM TTFT could be relatively large (in extreme cases 500–1200 ms or more, depending on model and load). • After (1k tokens): LLM TTFT can drop by roughly 300–800 ms compared to the long prompt. • ElevenLabs TTS TTFB remains ~100–300 ms (example), so overall TTFA improves by approximately the same 300–800 ms reduction you get on TTFT. How to get a precise measurement for your setup 1. Instrument and capture: ◦ Measure LLM TTFT (time from sending request to first token received). ◦ Measure TTS TTFB (time from sending text to first audio byte). ◦ Measure end-to-end TTFA (end of user speech → first audible sample). 1. Compare before/after shortening the prompt to quantify the real-world impact in your environment. If you want, I can help with next steps: • Suggest a minimal instrumentation snippet you can add to capture TTFT / TTFB metrics. • Recommend which OpenAI models to test first for lower TTFT. • Walk through interpreting the measurements once you have them. Which would you prefer — a small instrumentation example to measure TTFT/TTFB, or model recommendations to test? Sources: LLM Metrics | LiveKit Documentation | Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | examples/frontdesk | livekit/livekit_composite | DeepWiki | Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit