LiveKit Community

how to use preemptive_generation

if my starting point is

await session.start(
        agent=starting_agent,
        room=ctx.room,
    )

Thanks for reaching out, Tyler. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.


To use `preemptive_generation` starting from your `session.start` call, set `preemptive_generation=True` when creating your `AgentSession` instance. Then, start the session as usual with your agent and room parameters.

*Example:*

```session = AgentSession(
    vad=ctx.proc.userdata["vad"],
    llm=openai.LLM(model="gpt-4o-mini"),
    stt=deepgram.STT(model="nova-3", language="multi"),
    tts=openai.TTS(),
    turn_detection=MultilingualModel(),
    preemptive_generation=True,  # Enable preemptive generation
)

await session.start(agent=starting_agent, room=ctx.room)
```
With `preemptive_generation` enabled:

• The agent generates a reply as soon as the final transcript is received, before the user turn is fully committed.
• If you update the chat context or tools in `on_user_turn_completed`, the preemptive generation is cancelled and a new reply is generated.
• This reduces response latency by overlapping model inference with user audio.
Make sure to set `preemptive_generation=True` in the `AgentSession` constructor before calling `session.start` with your agent and room.
 Sources: <https://docs.livekit.io/agents/build/audio|Agent speech and audio | LiveKit Docs> | <https://github.com/livekit/agents/blob/fd648b6c9c2f577c39a395a36585f152edac542d/examples/voice_agents/langgraph_agent.py#L1-L71|examples/voice_agents/langgraph_agent.py>