I m using `AgentSession` with the turn detection model VAD+S LiveKit Community #ask-ai

I’m using `AgentSession` with the turn detection m...

important-judge-78504

09/15/2025, 9:01 PM

I’m using

AgentSession

with the turn detection model (VAD+STT fusion). Is there a way to access the turn boundary metadata it produces — e.g. start/end timestamps or utterance spans — so I can align recorded audio with the final transcripts? Right now I only see

UserInputTranscribedEvent

with

is_final=True

, but I’m not sure if that’s the best/only signal for utterance segmentation.

tall-airline-16756

09/15/2025, 9:01 PM

Thanks for reaching out, Ed. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

tall-airline-16756

09/15/2025, 9:02 PM

I'm happy to help you access turn boundary metadata in

AgentSession

when using the turn detection model (VAD+STT fusion). Here are a couple of ways you can do this: • Turn Start/End Timestamps: You can use the

on_end_of_turn

callback through the

RecognitionHooks

interface. This will give you useful metadata like the end-of-speech timestamp (

last_speaking_time

) and the final transcript details. • Per-Word Timing: By enabling

use_tts_aligned_transcript=True

in your session configuration and implementing the

transcription_node

method, you’ll receive streamed

TimedString

objects that include start and end times for each word or phrase. If you'd like, I’d be more than happy to walk you through how to set up either of these methods in detail. Just let me know! Sources: livekit-plugins/livekit-plugins-turn-detector/README.md | Text and transcriptions | LiveKit Docs

important-judge-78504

09/15/2025, 9:04 PM

are those for the user or the agent? im trying to get turn detection start and stop timestamps for the audio data going to the STT

tall-airline-16756

09/15/2025, 9:05 PM

To help you capture the turn detection start and stop timestamps for the audio data sent to STT, here are some useful methods you can try: • Start Timestamp: You can use the

on_start_of_speech

callback within the

RecognitionHooks

interface. This callback activates when the user transitions from 'listening' to 'speaking', so you can easily record the current time as the start of speech. • Stop Timestamp: Similarly, the

on_end_of_speech

callback in the same interface will trigger when the user stops speaking. The

last_speaking_time

field then gives you the adjusted end-of-speech timestamp, considering any silence detected by VAD or STT. If you'd like, I’d be happy to walk you through how to implement these callbacks in your session configuration. Just let me know! Sources: Turn detection and interruptions | LiveKit Docs | livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/experimental/realti... | AssemblyAI integration guide | LiveKit Docs

important-judge-78504

09/15/2025, 9:07 PM

what is on_start_of_speech? do you mean user_state_changed?

tall-airline-16756

09/15/2025, 9:07 PM

I'm happy to help clarify this for you! The best way to capture the start of speech in

AgentSession

is by using the

user_state_changed

event, rather than

on_start_of_speech

. This event fires whenever the user's state changes, including the moment they start speaking. So, when you see the

new_state

set to

"speaking"

, that's your signal that speech has begun. If you'd like, I can guide you through how to implement

user_state_changed

important-judge-78504

09/15/2025, 9:09 PM

is that event directly emitted from the turn detection?

Open in Slack

Previous Next