https://livekit.io logo
Join Slack
Powered by
I'm using UserInputTranscribedEvent to make me a t...
# ask-ai
l
I'm using UserInputTranscribedEvent to make me a transcript with time for each word, but the created_at timestamp is off and doesn't correspond to the actual speech start, there's a few seconds difference
t
Thanks for reaching out, David. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thank you for your question! The 
createdAt
timestamp in the 
UserInputTranscribedEvent
is meant to capture the exact moment when the transcription is created. Sometimes, there might be a little delay because of speech-to-text (STT) processing time, which is totally normal. To help you get the most accurate timing: • Try using the 
createdAt
field from events where 
isFinal: true
, as this reflects the complete utterance timing more reliably. • Make sure your system clocks are synced up with NTP to prevent any clock drift issues. • Keep an eye on STT latency metrics to get a sense of any delays between the speech and the transcription. If you're running tests in a simulation environment, a tolerance-based comparison function can be really handy to account for those expected delays. Please feel free to reach out if you'd like me to walk you through this or if you have any other questions—I’m here to help! Sources: Events and error handling | LiveKit Docs | livekit-plugins/livekit-plugins-ultravox/livekit/plugins/ultravox/realtime/... | Cartesia TTS integration guide | LiveKit Docs | ElevenLabs TTS integration guide | LiveKit Docs | agents/src/voice/room_io/room_io.ts
l
how do I get he STT latency ?
t
To help you retrieve STT latency metrics, you can subscribe to the 
metrics_collected
event from the STT component. This event gives you a 
STTMetrics
object with some really useful properties: • `duration`: This tells you how long the STT model took to create the transcript. • `audio_duration`: This shows the length of the audio input that was sent to the STT model. • `streamed`: This indicates whether the transcription was done in streaming mode. Here’s a friendly example in Python to get you started:
Copy code
from livekit.agents import metrics
from livekit.plugins import deepgram

stt = deepgram.STT()

@stt.on("metrics_collected")
def on_stt_metrics(metrics: metrics.STTMetrics):
    print(f"STT Duration: {metrics.duration} seconds")
    print(f"Audio Duration: {metrics.audio_duration} seconds")
    print(f"Streamed: {metrics.streamed}")
If you have any questions or need a hand with setting this up, please don’t hesitate to ask—I’m here to help! Sources: agents/src/metrics/usage_collector.ts | Logs, metrics, and telemetry | LiveKit Docs | Capturing metrics | LiveKit Docs | livekit-agents/livekit/agents/metrics/base.py
l
this doesn't give me the latency of the provided speech. Is there a heuristic I should use between timestamp of UserInputTranscribedEvent and the actual speech ?
t
I'd love to understand a bit more about how you're capturing the actual speech timestamp. Are you using a specific event or marker to indicate when the speech starts, or maybe relying on some external timing method? Knowing this will really help us figure out the best way to calculate the latency together.
l
for now I'm trying to know the timestamp of when the user spoke, and I'm using UserInputTranscribedEvent created_at, but it's not correct
t
To help you find out exactly when a user started speaking, here are some friendly methods you can try out: • VAD Event Timing: You can use the 
on_start_of_speech
callback in the VAD system. This gives you a neat way to estimate the start time by subtracting the 
speech_duration
from the current time. Here's a quick example:
Copy code
def on_start_of_speech(self, ev: vad.VADEvent):
    start_time = time.time() - ev.speech_duration
    print(f"User started speaking at: {start_time}")
• User State Change Events: Another approach is to subscribe to the 
user_state_changed
event. When the state switches to 
speaking
, that timestamp marks the beginning of speech. Here's how you might do it:
Copy code
@session.on("user_state_changed")
def on_user_state_changed(event):
    if event.new_state == "speaking":
        print(f"User started speaking at: {time.time()}")
• End of Turn Metrics: Lastly, you can tap into the 
on_end_of_turn
hook to get detailed timing info, like when the user last spoke and the full duration. If you'd like, I'm here to help you walk through implementing any of these methods—just let me know! Sources: tests/test_agent_session.py | Text and transcriptions | LiveKit Docs | Voice activity detection (VAD) parameters | LiveKit Docs
l
@session.on("user_state_changed") def on_user_state_changed(event): if event.new_state == "speaking": print(f"User started speaking at: {time.time()}") this is still not right, it's delayed, Is there a timestamp in the event new_state ?
t
The 
user_state_changed
event includes a 
created_at
timestamp (in Python) or 
createdAt
(in TypeScript/JavaScript), which shows the exact moment the state transition happened. This timestamp is really helpful to know exactly when the 
new_state
was entered. If you’re noticing any delays, it might be a good idea to check that your system clocks are synchronized using NTP to keep everything in sync. Please feel free to reach out if you’d like me to help you with this or if you have any other questions—I’m here for you! Sources: agents/src/voice/events.ts | livekit-agents/livekit/agents/voice/agent_session.py
l
the timestamp is still wrong looking at the timestamp of the recording, whereas the timestamp I get from on_conversation_item_added for the tts is perfect
t
I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!
l
give me the VAD-level Events
Open in Slack
PreviousNext
Powered by