If the user speaks, I want to generate reply right after it

Thanks — that’s clear. You want two behaviors: • If a user speaks, immediately generate a reply. • If nobody speaks within 3 seconds after the agent enters, send a delayed greeting. Below is a small, complete pattern you can use in your agent to do both. It cancels the delayed greeting when the user starts speaking and then immediately triggers reply generation for that user speech. Important points up front • Use

user_state_changed

(or whichever event your session emits when a participant starts speaking) to detect speech start. • Cancel the delayed greeting when speech starts. • Call

generate_reply()

(or your normal reply path) immediately when you detect user speech. Example implementation

import asyncio
from livekit.agents import Agent, UserStateChangedEvent

class ResponsiveAgent(Agent):
    async def on_enter(self):
        user_spoke = False
        delayed_task = None

        # Handler: when a user starts speaking, cancel delayed greeting and reply immediately
        async def handle_user_state_changed(ev: UserStateChangedEvent):
            nonlocal user_spoke, delayed_task
            if ev.new_state == "speaking":
                user_spoke = True
                # Cancel the delayed greeting if scheduled
                if delayed_task:
                    delayed_task.cancel()

                # Immediately generate a reply for the user's speech.
                # Option A: If you have STT->LLM pipeline that will trigger automatically,
                # you may not need to call generate_reply here. If not, call it directly.
                await self.session.generate_reply(
                    instructions="Respond to the user's recent message."
                )

        # Register the listener (use async wrapper if your event system expects sync handlers)
        # If .on expects sync handler, wrap as:
        def _sync_wrapper(ev):
            asyncio.create_task(handle_user_state_changed(ev))

        self.session.on("user_state_changed", _sync_wrapper)

        # Delayed greeting task (fires after 3s if nobody spoke)
        async def delayed_greet():
            try:
                await asyncio.sleep(3)
                if not user_spoke:
                    await self.session.generate_reply(
                        instructions="Hello — it seems quiet. How can I help?"
                    )
            except asyncio.CancelledError:
                # cancelled because user started speaking
                pass

        delayed_task = asyncio.create_task(delayed_greet())

Notes & adjustments • If your session already processes incoming audio into text and triggers the LLM automatically, you should not call

generate_reply()

twice. Instead, rely on that pipeline and only cancel the delayed greeting. • If you must pass the transcribed text to the LLM yourself, call

generate_reply()

(or your LLM call) with the transcription or context as soon as you receive it from STT. • If your event system delivers a

user_utterance

or STT result event (instead of

user_state_changed

), prefer that event because it carries the transcript and ensures you reply to actual content rather than just "started speaking". • Ensure your

user_away_timeout

is set appropriately for presence tracking, but keep the greeting timer independent (3s) so it triggers as you want. If you share whether you get a transcript event (STT) or only

user_state_changed

, I’ll adapt the code to either: • call

generate_reply()

with the transcript, or • let the existing STT→LLM pipeline handle the reply and only cancel the delayed greeting.

The issue here is that the on_enter is triggered when the session starts; however the session starts before the caller picks up the phone from the outbound agent. Meaning we will hit the delayed greet before caller picks up, and it will sound like the agent speaks instantly.

Thanks — that makes sense and good catch. You should not start the 3s timer in

on_enter

because the session starts before the outbound call is answered. Instead, start the delayed greeting only after you detect the call is actually picked up, and keep the immediate-reply logic tied to user speech events. Quick summary of the approach I'm recommending • Don’t run the delayed greet in

on_enter

. • Listen for a call-answer/connected event (for example

participant_connected

, a SIP participant state change, or a

callStatus

attribute update) that means the caller has picked up. • When you see that the caller is connected, start a cancellable 3s task to send the greeting. • Also listen for user-speech events (preferably a transcript event or

user_turn_completed

) and cancel the delayed task and immediately generate a reply when the user speaks. Example pattern (conceptual) 1. In `on_enter`: register listeners but do not start the 3s timer. 2. On call-picked-up event: ◦ create a task: wait 3s then send greeting if nobody spoke. 1. On user speech /

user_turn_completed

/ STT result: ◦ cancel the 3s task if running ◦ immediately generate a reply using the transcript (or let your STT→LLM pipeline handle it) Concise code sketch

import asyncio

class OutboundAwareAgent(Agent):
    async def on_enter(self):
        self.delayed_task = None
        self.user_spoke = False

        # Called when caller actually connects / answers the call
        def on_participant_connected(ev):
            # start the 3s delayed greeting after pickup
            async def delayed_greet():
                try:
                    await asyncio.sleep(3)
                    if not self.user_spoke:
                        await self.session.generate_reply(instructions="Hello — how can I help?")
                except asyncio.CancelledError:
                    pass
            self.delayed_task = asyncio.create_task(delayed_greet())

        # Called when user finishes speaking (or you get STT transcript)
        async def on_user_turn(ev):  # ev may include transcript
            self.user_spoke = True
            if self.delayed_task:
                self.delayed_task.cancel()
            # Use transcript if available, otherwise trigger normal reply flow
            transcript = getattr(ev, "transcript", None)
            if transcript:
                await self.session.generate_reply(instructions=transcript)
            else:
                await self.session.generate_reply(instructions="Respond to user's recent message.")

        # Register listeners (event names depend on your session)
        self.session.on("participant_connected", on_participant_connected)
        # prefer transcript or user-turn complete event if available
        self.session.on("user_turn_completed", lambda ev: asyncio.create_task(on_user_turn(ev)))

Notes and suggestions • Prefer a transcript or

user_turn_completed

event rather than

user_state_changed("speaking")

for immediate reply. That ensures you reply to actual content (and gives you the text to send to the LLM). • If your platform emits a SIP or participant attribute change (e.g.,

callStatus

), use that to detect pickup. The exact event name depends on your integration; look for

participant_connected

,

participant_updated

with a connected flag, or a SIP answer event. • If multiple callers/participants can join, scope the logic to the outbound callee participant so you don’t start the timer on other early connections. • Make sure to cancel and cleanup tasks on

on_exit

so no stray tasks run after the session ends. If you tell me which event your system emits when the outbound call is answered (e.g.,

participant_connected

,

participant_updated

with

callStatus: "answered"

, or a SIP-specific event), I’ll adapt the snippet to those exact event names and show how to extract the transcript when generating the immediate reply. Sources: Pipeline nodes and hooks | LiveKit Documentation

I use the standard STT -> LLM -> TTS model

Thanks — that’s helpful. Since you’re using the standard STT → LLM → TTS pipeline, the simplest, most reliable approach is: • Don’t start the 3s delayed greeting in

on_enter

. • Start the 3s timer only after you detect the outbound call has actually been answered (participant connected / SIP answered event). • Let the STT→LLM pipeline drive replies when a transcript arrives: cancel the delayed greeting on transcript and trigger the normal reply flow immediately. Below is a practical pattern you can use. It: 1) waits for a call-picked-up event to start the 3s timer, 2) cancels that timer if a transcript arrives, and 3) lets your STT→LLM→TTS pipeline generate a reply immediately when you receive the transcript. Code sketch (adapt to your event names and session API):

import asyncio
from livekit.agents import Agent

class OutboundSTTAgent(Agent):
    async def on_enter(self):
        self.delayed_task = None
        self.user_spoke = False

        # 1) Call answered / participant connected handler
        def on_participant_connected(ev):
            # Only start the greeting timer for the callee / outbound target
            if not self._is_target_participant(ev):
                return

            async def delayed_greet():
                try:
                    await asyncio.sleep(3)        # wait 3s after pickup
                    if not self.user_spoke:
                        # Use the same generate_reply path as your STT->LLM pipeline
                        await self.session.generate_reply(
                            instructions="Hello — how can I help today?"
                        )
                except asyncio.CancelledError:
                    pass

            # Start (or restart) the delayed greeting task
            if self.delayed_task:
                self.delayed_task.cancel()
            self.delayed_task = asyncio.create_task(delayed_greet())

        # 2) STT transcript / user-turn-completed handler
        async def on_transcript(ev):
            # ev should include which participant and transcript text
            # Mark that the user spoke and cancel greeting
            self.user_spoke = True
            if self.delayed_task:
                self.delayed_task.cancel()

            # Let your normal STT->LLM pipeline produce the reply.
            # If your pipeline automatically processes the transcript, do nothing here.
            # Otherwise, explicitly send the transcript to generate_reply:
            transcript = getattr(ev, "transcript", None)
            if transcript:
                await self.session.generate_reply(instructions=transcript)
            # If your system automatically triggers the LLM on transcript,
            # remove the explicit generate_reply above to avoid double responses.

        # Register listeners (adjust event names to your system)
        self.session.on("participant_connected", on_participant_connected)
        # Prefer the transcript or user-turn-complete event
        # Use an async wrapper if .on expects sync handlers:
        self.session.on("transcript", lambda ev: asyncio.create_task(on_transcript(ev)))

    def _is_target_participant(self, ev):
        # Replace this with logic to identify the outbound callee participant
        # e.g., check ev.participant.role, ev.participant.sid, or ev.participant.is_sip
        return True

Notes and practical tips • Use the transcript (STT result) event rather than "started speaking" events. Transcript ensures you reply to actual content and gives you text to send to the LLM. • If your platform emits a

participant_updated

with a SIP

callStatus

or a specific "answered" event, use that to detect pickup. Tell me the exact event name you receive and I’ll adapt the snippet. • Ensure you only start the delayed greeting for the outbound callee (not other participants or early MCU/system participants). • Clean up/cancel

self.delayed_task

in

on_exit

so no stray tasks run after the session ends. • If your STT->LLM pipeline automatically calls the LLM on transcript, do not call

generate_reply()

a second time—just cancel the delayed task and let the automatic flow run. Would you share the exact event your integration emits when the callee picks up (for example

participant_connected

,

participant_updated

with

callStatus: "answered"

, or a SIP answer event)? I’ll produce a copy of the snippet tailored to that event and show how to scope it to the callee participant.