My livekit agent is not detecting the first utterance that is being said by the other agent. Or it sometimes detects it but only the latter part of the utterance is there a cold start or something that is causing this?

Thanks for reaching out, Rohit. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

We run a LiveKit Agents digital human against a customer voice agent; the remote leg is often SIP, not always another agent. We don’t set

participant_kinds

to include

PARTICIPANT_KIND_AGENT

.

session.start

only adds BVC on audio input. Issue: Right after pickup, the first remote utterance is sometimes missing or only the tail in finals — e.g. full IVR “Press 1 to reach customer support” but grading saw “support.” We only handle

user_input_transcribed

with

is_final=True

. We also wait ~5s after answer before our side speaks on agent-first flows. Config (English): Deepgram Flux (

flux-general-multi

,

eager_eot_threshold=0.3

),

turn_detection="stt"

, endpointing ~0.5s min / +4s max, Silero VAD (activation_threshold 0.5, min_speech_duration 0.05). Enriched playback:

preemptive_generation=False

. Plan: [Build vs paid — fill in] For SIP remotes, is

PARTICIPANT_KIND_AGENT

still the right lever, or should we tune Flux eager EOT / endpointing / VAD? Any known BVC + VAD + STT turn detection issue clipping the start of the first utterance? When the remote is a LiveKit agent, what

participant_kinds

/

RoomOptions

do you recommend?

Thanks — this is helpful detail. I’ll address your three questions and ask a couple quick clarifying items so I can give a precise recommendation. Quick answers • For SIP remotes: including

PARTICIPANT_KIND_AGENT

is not required for SIP; tuning Flux endpointing, VAD, and STT turn detection is the primary lever for missed first words on SIP legs. • For agent-to-agent remotes: you should set

RoomInputOptions.participant_kinds=[rtc.ParticipantKind.PARTICIPANT_KIND_AGENT]

so agents can receive/process other agents’ audio. • Known interactions: cold starts (on Build/free) and VAD/STT endpointing can clip the start of an utterance. BVC audio injection plus VAD/STT-based turn detection can drop the very beginning if the audio pipeline or detectors haven’t fully activated. Recommendations to try next 1. Reduce risk from cold-starts ◦ Confirm whether you’re on Build (free). If yes, cold starts can add several seconds of initialization—keep an agent session warm during tests or try a paid environment to compare. 1. Tune endpointing / Deepgram Flux ◦ Lower aggressive eager EOT behavior to avoid premature finals. Try increasing

eager_eot_threshold

(less eager) or adjust Flux model settings to be slightly less aggressive. ◦ Consider increasing the minimum endpointing (e.g., from 0.5s to ~0.7–1.0s) to give upstream audio/VAD time to stabilize. 1. Tune VAD and STT turn detection ◦ Increase VAD warmup tolerance: raise

min_speech_duration

slightly (from 0.05s to 0.1–0.15s) so very short initial frames don’t get ignored, and ensure

activation_threshold

is not too high for the remote audio level. ◦ For

turn_detection="stt"

, extend endpointing tail (max) slightly or use a slightly longer EOT buffer so STT has time to decide, especially for noisy SIP legs. 1. BVC + playback specifics ◦ If you inject BVC on audio input at

session.start

, ensure the BVC stream is fully established before the remote begins speaking; otherwise the initial packets can be missed. ◦ Keep

preemptive_generation=False

if you want generation only after stable input; but when first-utterance loss is the concern, test with controlled pre-roll (delay outbound speak) to confirm detection timing. 1. Test matrix (to isolate cause) ◦ Test A: SIP remote → keep receiving agent warmed 5–10s before remote speaks; keep current settings. ◦ Test B: SIP remote → increase VAD

min_speech_duration

to 0.1–0.15s and raise `eager_eot_threshold`; compare. ◦ Test C: LiveKit agent remote → enable

participant_kinds

for agents and repeat above tests. ◦ Compare results across Build vs paid environments to surface cold-start effects. Configuration for LiveKit-agent remotes • Use

RoomInputOptions(participant_kinds=[rtc.ParticipantKind.PARTICIPANT_KIND_AGENT])

. • Keep similar VAD/STT tuning as above; agent-to-agent audio still needs VAD/STT stability and may be affected by cold starts if sessions aren’t warm. One more helpful data point • Could you confirm whether the missing-first-word behavior occurs only immediately after pickup/answer, or also mid-call? Also, which plan are you on (Build/free or paid)? With that I can recommend exact values for

eager_eot_threshold

, endpointing min/max, and VAD params to try., with interactive elements

Our setup is a LiveKit Agents digital human against a customer voice agent; the remote leg is often SIP, not always another agent. We are on the paid plan, not Build/free. The missed or partial first utterance shows up most often right after pickup or answer, especially on the first IVR or opening line. We have also seen short finals mid-call, but the worst cases are early in the call. We wait about 5 seconds after answer before our side speaks on agent-first flows.

Thanks — that context is perfect. I’ll focus on what you can change now given you’re on paid plan, SIP remotes are common, and the problem is concentrated right after pickup. Short summary of likely causes • Cold-starts are not the culprit (you’re on paid), so missed starts are most likely due to a combination of early STT endpointing (Deepgram Flux), VAD activation/warm‑up, and BVC/audio pipeline timing right after answer. • For SIP remotes, setting

PARTICIPANT_KIND_AGENT

is not required; tuning Flux / endpointing / VAD is the primary lever. For LiveKit-agent remotes, enable agent participant kinds so agents can receive/process agent audio. Actionable recommendations (ordered by impact) 1. Give STT/VAD a little more breathing room right after answer ◦ Increase minimum endpointing to ~0.7–1.0s (from 0.5s) so transient startup audio isn’t finalized too quickly. ◦ Increase the EOT tail (max) slightly (e.g., +0.5–1.0s) to avoid premature finalization on short starts. 1. Make Flux less aggressive on eager EOT ◦ Try increasing

eager_eot_threshold

(less eager). If currently 0.3, test values like 0.45–0.6 and compare. ◦ Monitor whether finals include the whole IVR phrase rather than just the tail. 1. Soften VAD activation/warm-up ◦ Raise

min_speech_duration

from 0.05s to ~0.10–0.15s so the VAD requires slightly longer contiguous voice before activating. ◦ Confirm

activation_threshold

is not too high for SIP audio levels; try modestly lower values if audio is distant/attenuated. 1. Stabilize BVC/audio pipeline at call start ◦ Ensure your BVC injection is fully established before the remote’s first audio is expected. Delay any BVC playback or generation until you’ve observed a stable incoming audio buffer for ~300–500ms. ◦ If you can’t delay playback, add a short receive-warmup on the agent side (listen-only for ~300–700ms) before running STT/turn detection. 1. Turn-detection tuning ◦ With

turn_detection="stt"

, favor slightly longer detection windows at call start. Increase EOT thresholds and STT buffer durations for the first N seconds after answer (you can make this adaptive: longer windows for first 5–10s, then normal afterwards). 1. Test matrix (quick, controlled tests) ◦ Test 1: Increase endpointing min to 0.8s,

eager_eot_threshold

→ 0.5, VAD

min_speech_duration

→ 0.12s. Reproduce with IVR. ◦ Test 2: Add a 300–500ms receive-warmup before processing STT after answer (no other changes). ◦ Test 3: For agent remotes, enable

participant_kinds=[rtc.ParticipantKind.PARTICIPANT_KIND_AGENT]

and repeat tests to confirm parity. ◦ Compare detection/final transcript completeness across tests. Suggested settings to try immediately • Flux:

eager_eot_threshold=0.45–0.6

• Endpointing:

min_endpoint=0.7–1.0s

,

max_endpoint += 0.5–1.0s

• VAD:

min_speech_duration=0.10–0.15s

, slightly lower

activation_threshold

if SIP audio is soft • First-utterance handling: apply a +300–700ms STT/VAD warmup window after answer (adaptive shorter windows after first 5–10s) When to set

PARTICIPANT_KIND_AGENT

• If the remote is a LiveKit agent, set

RoomInputOptions.participant_kinds=[rtc.ParticipantKind.PARTICIPANT_KIND_AGENT]

so agents receive/process agent audio. • For SIP remotes this is unnecessary and tuning Flux/VAD/STT is the correct path. If you want, I can: • Propose an exact set of parameter values to run A/B tests on IVR openers you’ve seen clipped. • Help you craft a short test script (sequence and timings) to reproduce the issue consistently. Which would you prefer, and do you want me to propose the exact A/B parameter sets to run first? , with interactive elements