https://livekit.io logo
Join Slack
Powered by
I'd like to get the transcriptions for both the us...
# ask-ai
f
I'd like to get the transcriptions for both the user and agent during a session. Along with the transcription I'd like to get the time in which an utterance was started and ended by either participant. I'm currently making use of the user_state_changed, agent_state_changed, and conversation_item_added events, but often times the "speaking" event for user_state_changed is not recognized or it's recognized and not accompanied by a transcription from "conversation_item_added". What would be the best way of getting this transcription information?
t
Thanks for reaching out, Abdullah. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
To help you get transcriptions with start and end times for utterances, here’s a friendly step-by-step approach you can follow: 1. Enable TTS-Aligned Transcriptions: In Python, simply set 
use_tts_aligned_transcript=True
in your 
AgentSession
. This will make sure your speech and transcription are perfectly synchronized, including all the timing details. 2. Implement the 
transcription_node
Method: This method gives you access to 
TimedString
objects, which carry the 
start_time
and 
end_time
for each utterance—super handy! 3. Access Transcriptions on the Frontend: You can use LiveKit SDKs to subscribe to transcription events through the 
lk.transcription
text stream topic. The 
TranscriptionSegment
type includes 
startTime
and 
endTime
fields for your convenience. 4. Choose a TTS Provider: Providers like Cartesia or ElevenLabs offer word-level timing. If you prefer sentence-level timing, OpenAI TTS works well, though you might need to infer the 
end_time
. If you’d like, I’d be more than happy to help you set up a specific TTS provider or guide you through accessing transcriptions in the SDK you’re working with. Just let me know! Sources: Text and transcriptions | LiveKit Docs | Text and transcriptions | LiveKit Docs | Understanding Transcript Timing in Voice Conversations | livekit-plugins/livekit-plugins-cartesia/livekit/plugins/cartesia/stt.py | transcription.go
f
I understand that use_tts_aligned_transcript may help me with agent speech, but will it get the start and end times for user speech?
t
To help you find the start and end times for user speech transcriptions, you can use the 
user_input_transcribed
event. This event includes transcript segments along with timing details like 
start_time
and 
end_time
, which come straight from the STT provider's response. Just a quick note: make sure your STT provider supports timing metadata. For instance: • Baseten STT: Pulls 
start
and 
end
times directly from the transcription response. • ElevenLabs STT: Calculates 
start_time
and 
end_time
using word-level timestamps. If you'd like, I’m happy to walk you through setting up a specific STT provider or help you access these events in your setup. Just let me know! Sources: lib/src/proto/livekit_models.pb.dart | Text and transcriptions | LiveKit Docs | packages/livekit-rtc/src/proto/room_pb.ts | Sources/LiveKit/Protos/livekit_models.pb.swift | Understanding Transcript Timing in Voice Conversations
f
From the current docs: user_input_transcribed A 
UserInputTranscribedEvent
is emitted when user transcription is available. Properties • `language`: str • `transcript`: str • `is_final`: bool • `speaker_id`: str | None - Only available if speaker diarization is supported in your STT plugin. user_input transcribed does not have the relevant stt info I need, which is the start and end time of the user's utterance. How can I get that information?
t
If the 
user_input_transcribed
event doesn’t provide start and end times, don’t worry! You can still get this information by trying a couple of helpful methods: 1. Use an STT Provider with Timing Support: Many STT providers, like Deepgram, Google, and Baseten, include 
start_time
and 
end_time
in their transcription responses. Just make sure your STT plugin is set up to extract these timestamps. 2. Enable VAD-Based Endpointing: If your STT provider doesn’t support timing, you can use Voice Activity Detection (VAD) to figure out when speech starts and ends. You can configure your 
AgentSession
with 
turn_detection="vad"
and use a VAD tool like Silero to detect the speech segments. If you’d like, I’m here to help you set up VAD or check your STT provider’s capabilities. Just let me know! Sources: livekit-plugins/livekit-plugins-gladia/livekit/plugins/gladia/stt.py | Turn detection and interruptions | LiveKit Docs | OpenAI Realtime API integration guide | LiveKit Docs | Understanding Transcript Timing in Voice Conversations
f
Is it possible to set the tts config to send sentence level timing instead of word level timing?
t
Hi there! When using TTS providers like Cartesia and ElevenLabs, turning on 
use_tts_aligned_transcript=True
will give you word-level timing by default. At the moment, there isn’t an option to switch these providers to sentence-level timing, but I’m here to help you navigate this! On the other hand, if you’re working with other TTS providers like OpenAI TTS, enabling 
use_tts_aligned_transcript=True
will provide sentence-level timing. Plus, sentence-level timing is also available in non-streaming TTS setups using the StreamAdapter. If you'd like, I’d be happy to help you explore providers that support sentence-level timing or assist with configuring your current setup to best meet your needs. Just let me know! Sources: Text and transcriptions | LiveKit Docs | Text and transcriptions | LiveKit Docs | Docs::Agents | livekit-agents/livekit/agents/voice/transcription/synchronizer.py
4 Views
Open in Slack
PreviousNext
Powered by