need help with manual turn detection

Thanks for reaching out, Lawrence. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

im using manual turn detection, with realtime pipeline. nodejs. in that case, the livekit agent will not have to process speech to text before sending audio stream to the realtime llm right?

You're correct — with manual turn detection + a realtime model in Node.js, the LiveKit agent does not need to run STT before sending audio to the realtime LLM. Key points: • Set

turnDetection: 'manual'

and control turn boundaries with

commitUserTurn()

(or your JS equivalent). • The agent buffers raw audio and forwards it to the realtime model when you commit the turn. • The realtime model performs any internal speech recognition and response generation; the agent doesn't transcribe unless you explicitly add an STT plugin. • Add STT only if you need live transcripts or want to use the LiveKit turn-detector model (which requires STT output). If you want, share a short code snippet of how you're enabling/disabling audio and calling

commitUserTurn()

and I’ll check it for issues. Sources: Realtime model plugins | LiveKit Docs | Improving voice AI's turn detection with transformers | Turns overview | LiveKit docs

im looking at the audio_recognition.ts code. i see this “”"*commitUserTurn*(_audioDetached_: boolean) { const commitUserTurnTask = (_delayDuration_: number = 500) => async (_controller_: AbortController) => { if (Date.now() - this.lastFinalTranscriptTime > delayDuration) { // flush the stt by pushing silence if (audioDetached && this.sampleRate !== undefined) { const numSamples = Math.floor(this.sampleRate * 0.5); const silence = new Int16Array(numSamples * 2); const silenceFrame = new AudioFrame(silence, this.sampleRate, 1, numSamples); this.silenceAudioWriter.write(silenceFrame); } // wait for the final transcript to be available await delay(delayDuration, { signal: controller.signal }); } if (this.audioInterimTranscript) { // append interim transcript in case the final transcript is not ready this.audioTranscript =

${this.audioTranscript} ${this.audioInterimTranscript}

.trim(); } this.audioInterimTranscript = ‘’; const chatCtx = this.hooks.retrieveChatCtx(); this.logger.debug(‘running EOU detection on commitUserTurn’); this.runEOUDetection(chatCtx); this.userTurnCommitted = true; }; // cancel any existing commit user turn task this.commitUserTurnTask?.cancel(); this.commitUserTurnTask = Task.from(commitUserTurnTask()); this.commitUserTurnTask.result .then(() => { this.logger.debug(‘User turn committed’); }) .catch((_err_: unknown) => { this.logger.error(err, ‘Error in user turn commit task:’); }); } async *close*() { this.detachInputAudioStream(); this.silenceAudioWriter.releaseLock(); await this.commitUserTurnTask?.cancelAndWait(); await this.sttTask?.cancelAndWait(); await this.vadTask?.cancelAndWait(); await this.bounceEOUTask?.cancelAndWait(); }“”", it seems to be doing some waiting on speech to text transcript? am i right?

You're reading it correctly — that code waits for STT finalization when STT is enabled. What the snippet does: • If enough time has passed since the last final transcript, it optionally writes a short silence frame (when

audioDetached

and

sampleRate

are set) to flush STT buffering, then awaits a small delay so a final transcript can arrive. • It appends any

audioInterimTranscript

to the final

audioTranscript

, clears the interim, runs end‑of‑utterance (EOU) detection, and marks the user turn committed. • The commit task is cancellable and raced with other tasks;

close()

cancels/waits for those tasks. How this relates to realtime models / your setup: • This waiting only matters if you have STT enabled (or are using the LiveKit turn‑detector which consumes STT). With a realtime audio model and manual turn detection (no STT plugin), the agent won’t be producing transcripts, so that STT‑waiting behavior is effectively a no‑op — the agent simply buffers audio and forwards it when you call

commitUserTurn()

. • If you see delays and you do not need transcripts, ensure STT is disabled or not configured so the commit path doesn’t wait for transcript finalization. If you want, share how you’ve configured STT/realtime in Node.js (small config snippet) and I’ll point out exactly which flags to change.

that’s from the livekit agent sdk code. doesn’t “if (Date.now() - this.lastFinalTranscriptTime > delayDuration) {” mean that it’s going to always call the delay function??

Good question — that condition does not always call the delay. Here's what it means and when it runs: • The

if (Date.now() - this.lastFinalTranscriptTime > delayDuration)

checks how long it’s been since the last final STT transcript arrived. • The delay block runs only when that elapsed time is greater than

delayDuration

(default 500 ms). In other words, it waits only if a final transcript hasn’t been received recently — it tries to flush STT buffering by writing a short silence (when

audioDetached

) and then awaits the delay so a final transcript can arrive. • If a final transcript arrived within

delayDuration

, the

if

condition is false and the delay/silence path is skipped. Practical note for your setup: • With manual turn detection + a realtime model and no STT plugin, there are no transcripts, so

lastFinalTranscriptTime

is likely unchanged and that block effectively does nothing relevant (the agent just buffers audio and commits immediately). If you’re seeing unexpected waits, share

lastFinalTranscriptTime

updates or your STT config and I’ll help pinpoint why.

it does cause the delay though. i just tested it. i dont have speech to text pipeline, yet it caused that delay function to be triggered. here’s my setup: “”"private async *initialize*(): Promise<void> { const session = new voice.AgentSession({ llm: new ManualRealtimeModel({ model: ‘gpt-realtime-mini’, turnDetection: null, modalities: [‘audio’, ‘text’], }), turnDetection: ‘manual’, voiceOptions: { preemptiveGeneration: false, minEndpointingDelay: 0, maxEndpointingDelay: 0, minInterruptionDuration: 0, allowInterruptions: false, }, }); session.on(voice.AgentSessionEventTypes.MetricsCollected, (ev) => { metrics.logMetrics(ev.metrics); this.usageCollector.collect(ev.metrics); }); session.on(voice.AgentSessionEventTypes.ConversationItemAdded, this.onConversationItemAdded); await session.start({ agent: this.agent, room: this.room, inputOptions: { audioEnabled: true, textEnabled: false, }, outputOptions: { audioEnabled: false, transcriptionEnabled: true, }, }); session.input.setAudioEnabled(false); this.session = session; }“”"

Thanks — I looked at your setup and the cause is clear. Why you still hit the delay • You started the session with

outputOptions.transcriptionEnabled: true

. That enables the SDK’s transcription flow (so

lastFinalTranscriptTime

is never updated with a recent final transcript), which makes the

if (Date.now() - this.lastFinalTranscriptTime > delayDuration)

branch true and triggers the silence+delay even though you don’t have an external STT pipeline. • In short: transcription is enabled, so the commit task waits for a final transcript (or the delay) even when you expect no STT. How to avoid the delay • Turn off transcription if you don’t want any STT behavior: ◦ Set

outputOptions.transcriptionEnabled: false

when calling

session.start(...)

. • Or, if you need transcriptionEnabled for other reasons, you can: ◦ Reduce

delayDuration

where commitUserTurn is called (not recommended unless you modify SDK code). ◦ Ensure

lastFinalTranscriptTime

is updated earlier by wiring a lightweight STT or interim finalization path. Recommended change for your case • Update your start call to: ◦

outputOptions: { audioEnabled: false, transcriptionEnabled: false }

This will stop the SDK from waiting for STT final transcripts and remove the delay. If you want, I can: • Point to the exact flag in your SDK call to change, or • Review a short snippet after you change

transcriptionEnabled

to confirm the delay is gone.

the issue is still there. i already changed transcriptionEnabled to false

Thanks — I hear your frustration. You changed

transcriptionEnabled

to

false

and the delay still happens; let’s narrow this down. Quick checklist so I can pinpoint the cause — please reply with these items: 1. SDK version (the exact package version for the LiveKit Agent SDK you’re using). 2. Are you loading any STT/transcription plugin elsewhere (explicitly or via a wrapper)? Paste any plugin-related code or session plugins list. 3. The exact call site where you call

session.commitUserTurn()

(or

commitUserTurn()

flow). Are you calling it directly, or relying on the SDK task created by

commitUserTurn()

? 4. A short excerpt of logs around the commit path showing timestamps, including values for

lastFinalTranscriptTime

if possible. Example log lines or timestamps before/after commit help confirm which branch runs. 5. Confirm

outputOptions

you currently pass to

session.start()

(full object) and whether

inputOptions.audioEnabled

is toggled before/after commit. Quick troubleshooting steps you can try now: • Temporarily set

minEndpointingDelay

and

maxEndpointingDelay

in

voiceOptions

to a nonzero small value (e.g., 50) to see if behavior changes. • Search your app for any code that might call

session.input.setTranscriptionEnabled(true)

or similar after start. • Add a debug log right before

commitUserTurn()

that prints

Date.now()

and

this.lastFinalTranscriptTime

(or listen for the SDK event that exposes it) to verify the condition. If you send the items above (especially SDK version + commit call snippet + logs), I’ll point to the exact code path and a specific fix.

1. “@alert-easter-19258/agents”: “1.0.30", “@alert-easter-19258/agents-plugin-livekit”: “1.0.30", “@alert-easter-19258/agents-plugin-openai”: “1.0.30", 2. no 3. “”"*startTurn* = async (): Promise<string> => { const session = this.getSession(); if (this.manualTurnActive) { utils.log(‘startTurn ignored because a turn is already active’); return ‘error’; } this.manualTurnActive = true; this.turnStartTime = Date.now(); session.interrupt(); session.clearUserTurn(); this.setAudioEnabled(true); utils.log(‘startTurn command received: opening turn’); return ‘ok’; }; endTurn = async () => { const session = this.getSession(); const endTurnReceivedAt = Date.now(); if (!this.manualTurnActive) { utils.log(‘endTurn ignored because no turn is active’); return ‘error’; } this.manualTurnActive = false; this.setAudioEnabled(false); const turnDuration = endTurnReceivedAt - this.turnStartTime; if (turnDuration < this.MIN_TURN_DURATION_MS) { utils.log(

endTurn received but turn duration (${turnDuration}ms) is too short; clearing turn without commit

, ); session.clearUserTurn(); return ‘error’; } utils.log(‘endTurn committing turn’); session.commitUserTurn(); utils.log(‘endTurn user turn committed, waiting for grading response’); return ‘ok’; };“”" 4. “”"[2025-12-19T112320.914Z] HayaiAgent: startTurn command received: opening turn [182320.533] DEBUG (lk-rtc/53799): Connect callback received [2025-12-19T112323.028Z] HayaiAgent: endTurn committing turn ts2025 12 19T1123:23.028Z commitUserTurnTask begin !!!! ts: 2025-12-19T112323.029Z waiting for the final transcript to be available !!!! [2025-12-19T112323.029Z] HayaiAgent: endTurn user turn committed, waiting for grading response ts: 2025-12-19T112323.531Z final transcript is available !!!! [182323.534] INFO (53799): Creating speech handle speech_id: “speech_4a012107-551" [182323.537] INFO (53799): EOU metrics endOfUtteranceDelayMs: 0 transcriptionDelayMs: 0 onUserTurnCompletedDelayMs: 0 [2025-12-19T112324.161Z] HayaiAgent: >>> transcriptionNode triggered [2025-12-19T112324.162Z] HayaiAgent: >>> transcription partial { [2025-12-19T112324.163Z] HayaiAgent: >>> transcription partial [2025-12-19T112324.373Z] HayaiAgent: >>> transcription partial ” [2025-12-19T112324.374Z] HayaiAgent: >>> transcription partial score [2025-12-19T112324.375Z] HayaiAgent: >>> transcription partial “: [2025-12-19T112324.376Z] HayaiAgent: >>> transcription partial [2025-12-19T112324.376Z] HayaiAgent: >>> transcription partial 70 [2025-12-19T112324.376Z] HayaiAgent: >>> transcription partial , [2025-12-19T112324.377Z] HayaiAgent: >>> transcription partial [2025-12-19T112324.377Z] HayaiAgent: >>> transcription partial [2025-12-19T112324.378Z] HayaiAgent: >>> transcription partial ” [2025-12-19T112324.386Z] HayaiAgent: >>> transcription partial feedback [2025-12-19T112324.591Z] HayaiAgent: >>> transcription partial “: [2025-12-19T112324.591Z] HayaiAgent: >>> transcription partial ” [2025-12-19T112324.592Z] HayaiAgent: >>> transcription partial Focus [2025-12-19T112324.592Z] HayaiAgent: >>> transcription partial on [2025-12-19T112324.598Z] HayaiAgent: >>> transcription partial the [2025-12-19T112324.598Z] HayaiAgent: >>> transcription partial correct [2025-12-19T112324.600Z] HayaiAgent: >>> transcription partial pitch [2025-12-19T112324.600Z] HayaiAgent: >>> transcription partial accent [2025-12-19T112324.601Z] HayaiAgent: >>> transcription partial , [2025-12-19T112324.601Z] HayaiAgent: >>> transcription partial particularly [2025-12-19T112324.862Z] HayaiAgent: >>> transcription partial on [2025-12-19T112324.863Z] HayaiAgent: >>> transcription partial ' [2025-12-19T112324.868Z] HayaiAgent: >>> transcription partial スーパー [2025-12-19T112324.868Z] HayaiAgent: >>> transcription partial ' [2025-12-19T112324.870Z] HayaiAgent: >>> transcription partial to [2025-12-19T112324.871Z] HayaiAgent: >>> transcription partial sound [2025-12-19T112324.871Z] HayaiAgent: >>> transcription partial more [2025-12-19T112324.871Z] HayaiAgent: >>> transcription partial natural [2025-12-19T112325.073Z] HayaiAgent: >>> transcription partial .“, [2025-12-19T112325.074Z] HayaiAgent: >>> transcription partial [2025-12-19T112325.074Z] HayaiAgent: >>> transcription partial ” [2025-12-19T112325.074Z] HayaiAgent: >>> transcription partial trans [2025-12-19T112325.074Z] HayaiAgent: >>> transcription partial cription [2025-12-19T112325.074Z] HayaiAgent: >>> transcription partial “: [2025-12-19T112325.075Z] HayaiAgent: >>> transcription partial ” [2025-12-19T112325.075Z] HayaiAgent: >>> transcription partial スーパー [2025-12-19T112325.075Z] HayaiAgent: >>> transcription partial の [2025-12-19T112325.076Z] HayaiAgent: >>> transcription partial 店 [2025-12-19T112325.105Z] HayaiAgent: >>> transcription partial 員 [2025-12-19T112325.105Z] HayaiAgent: >>> transcription partial さん [2025-12-19T112325.112Z] HayaiAgent: >>> transcription partial と [2025-12-19T112325.114Z] HayaiAgent: >>> transcription partial ” [2025-12-19T112325.116Z] HayaiAgent: >>> transcription partial }“”" 5. i also hacked audio_recongition.js in the livekit agent sdk to log timestamps and events “”"*commitUserTurn*(audioDetached) { var _a; const commitUserTurnTask = (delayDuration = 500) => async (controller) => { if (Date.now() - this.lastFinalTranscriptTime > delayDuration) { if (audioDetached && this.sampleRate !== void 0) { const numSamples = Math.floor(this.sampleRate * 0.5); const silence = new Int16Array(numSamples * 2); const silenceFrame = new AudioFrame(silence, this.sampleRate, 1, numSamples); this.silenceAudioWriter.write(silenceFrame); } console.log(

ts: ${new Date().toISOString()} waiting for the final transcript to be available !!!!

); await delay(delayDuration, { signal: controller.signal }); console.log(

ts: ${new Date().toISOString()} final transcript is available !!!!

); } if (this.audioInterimTranscript) { this.audioTranscript =

${this.audioTranscript} ${this.audioInterimTranscript}

.trim(); } this.audioInterimTranscript = “”; const chatCtx = this.hooks.retrieveChatCtx(); this.logger.debug(“running EOU detection on commitUserTurn”); this.runEOUDetection(chatCtx); this.userTurnCommitted = true; }; console.log(

ts:${new Date().toISOString()} commitUserTurnTask begin !!!!

); (_a = this.commitUserTurnTask) == null ? void 0 : _a.cancel(); this.commitUserTurnTask = Task.from(commitUserTurnTask()); this.commitUserTurnTask.result.then(() => { this.logger.debug(“User turn committed”); }).catch((err) => { this.logger.error(err, “Error in user turn commit task:“); }); }“”"

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!