https://livekit.io logo
Join Slack
Powered by
what is the best way to get timestamps for the sta...
# ask-ai
i
what is the best way to get timestamps for the start and end of a user utterance
t
Thanks for reaching out, Ed. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
I'm happy to help you get timestamps for the start and end of a user utterance in LiveKit! Here are some ways you can do that: • Speech-to-Text (STT) Plugins: Many popular STT plugins like Google STT, Baseten, and Sarvam include 
start_time
and 
end_time
in their transcription results. These timestamps usually come in seconds and mark the beginning and end of the recognized speech segment. • Python SDK: If you're working with the Python SDK, you can turn on 
use_tts_aligned_transcript=True
in the 
AgentSession
. Doing this lets the transcription iterator provide 
TimedString
objects that include 
start_time
and 
end_time
in seconds, relative to the agent's current turn. Just a heads-up, this feature is still experimental. • Protocol or Client SDK Models: The LiveKit protocol and client SDKs (like Swift or Flutter) offer fields such as 
startTimestampMs
and 
endTimestampMs
within transcription-related structures, which capture timing in milliseconds. • Voice Activity Detection (VAD): Another approach is to track the internal state when VAD detects speech presence or absence, helping you infer when user speech starts and ends. For the most consistent and reliable timing, I’d recommend using STT plugins since they provide timing metadata directly. If you'd like, I’m here to walk you through implementing any of these methods or answer any questions you have! Sources: Sources/LiveKit/Protos/livekit_models.pb.swift | livekit-plugins/livekit-plugins-google/livekit/plugins/google/stt.py | Text and transcriptions | LiveKit Docs | Text and transcriptions | LiveKit Docs | livekit/livekit_models.pb.go
i
we use deepgram, basically what i want to understand is for each audio utterance from the user, what are the start and end timestamps
t
To get start and end timestamps for user utterances using Deepgram in LiveKit, let's walk through these steps together: 1. Install and Configure Deepgram: First, install the Deepgram plugin for LiveKit Agents by running 
pip install "livekit-agents[deepgram]~=1.2"
. Make sure you have your Deepgram API key ready, either set as an environment variable or passed directly when initializing. 2. Initialize Deepgram STT: Next, create an instance of the Deepgram STT with your preferred settings, like 
endpointing_ms
which helps define the silence duration for speech endpointing:
Copy code
from livekit.plugins import deepgram

stt = deepgram.STT(
    model="nova-3",
    language="en-US",
    endpointing_ms=25
)
3. Retrieve Timestamps: You can then access the 
start_time
and 
end_time
from Deepgram's transcription results. Here's a simple example in Python:
Copy code
def live_transcription_to_speech_data(language: str, data: dict, *, is_final: bool):
    dg_alts = data["channel"]["alternatives"]
    for alt in dg_alts:
        start_time = alt["words"][0]["start"] if alt["words"] else 0
        end_time = alt["words"][-1]["end"] if alt["words"] else 0
        print(f"Start: {start_time}, End: {end_time}")
4. Integrate with AgentSession: Use your STT instance within the 
AgentSession
to manage user turn detection and transcription events smoothly:
Copy code
from livekit.agents import AgentSession

session = AgentSession(stt=stt)
5. Subscribe to Events: Finally, listen for transcription events like 
user_input_transcribed
to get timing metadata easily:
Copy code
@session.on("user_input_transcribed")
def on_transcript(event):
    for alt in event.alternatives:
        print(f"Start: {alt.start_time}, End: {alt.end_time}, Text: {alt.text}")
If you have any questions or need a hand with the setup or configuration, please don't hesitate to reach out. I'm here to help! Sources: plugins/deepgram/src/stt.ts | Speech-to-text (STT) integrations | LiveKit Docs | Deepgram STT integration guide | LiveKit Docs | Deepgram STT integration guide | LiveKit Docs | livekit-agents/livekit/agents/voice/audio_recognition.py
5 Views
Open in Slack
PreviousNext
Powered by