https://livekit.io logo
Join Slack
Powered by
what is preemptive_generation in Livekit agent
# ask-ai
l
what is preemptive_generation in Livekit agent
t
Thanks for reaching out, Deepak. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’ll explain. Preemptive_generation is a LiveKit Agent option that starts LLM inference and TTS synthesis on interim (partial) STT transcripts before the user finishes speaking. It speculatively prepares a response so the agent can reply faster when the user’s turn ends. How it works • When 
preemptive_generation=True
(AgentSession), the agent uses interim transcripts to begin generating a response and may start TTS. • It only triggers if speech playback can be interrupted, the agent isn’t draining, and a standard LLM (not a RealtimeModel) is used. • If the final transcript matches the interim transcript closely, the preemptive result is reused; if the final transcript differs or the user interrupts, the preemptive output is discarded and a new response is generated. Benefits • Reduces perceived latency and enables more natural, low-latency conversations. Tradeoffs • Uses extra compute for speculative generations that may be canceled. If you want, I can show the exact code snippet for your language (Python or Node.js) or help decide whether to enable it for your agent. Which would you prefer? Sources: examples/voice_agents | livekit/livekit_composite | DeepWiki | Agent speech and audio | LiveKit docs | Pipeline Translator Agent | LiveKit docs | Agent session | LiveKit docs | Agents (Python) | livekit/livekit_composite | DeepWiki
l
yes
t
Great — which language would you like the code snippet and setup for: Python or Node.js? Feel free to also share your current AgentSession config if you want a tailored example.
l
python
t
Below is a concise Python example showing how to enable and use 
preemptive_generation
in an AgentSession, plus brief notes on behavior and tuning.
Copy code
from livekit_voice import AgentSession
from livekit_voice.integrations import deepgram, openai, elevenlabs

# Example: create an AgentSession with preemptive generation enabled
session = AgentSession(
    stt=deepgram.STT(api_key="DEEPGRAM_API_KEY"),
    llm=openai.LLM(api_key="OPENAI_API_KEY"),     # standard LLM (not RealtimeModel)
    tts=elevenlabs.TTS(api_key="ELEVENLABS_API_KEY"),
    vad=None,                                     # include your VAD if used
    preemptive_generation=True                     # <-- enable preemptive generation
)

# Start the session / run loop (pseudo-flow; integrate with your audio input pipeline)
session.start()

# When interim transcripts arrive, the agent may begin LLM + TTS generation automatically.
# When final transcript arrives, preemptive output will be reused if it matches, otherwise discarded.
# You can stop the session when done:
# session.stop()
Notes and tips • Ensure you use a standard LLM (RealtimeModel is not supported for preemptive generation). • Preemptive generation triggers only when playback is interruptible and the agent is not draining. • Expect increased speculative compute since partial work may be canceled if the final transcript differs or the user interrupts. • If you want, share your current AgentSession settings and I can suggest VAD, interruption policy, or cost-control tweaks.
3 Views
Open in Slack
PreviousNext
Powered by