Hi! I added exclude timestamp = False on the session.history.to_dict(exclude_timestamp=False) Now, the thing here is that the transcript now has a 'created at' field, but I notice it's wrong. The timing is actually wrong. So what's going on with that?

Thanks for reaching out, Sebastian. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

In the sense that it doesn't map correctly to the audio file.

Thanks so much for your question! The 'created_at' field in

session.history.to_dict(exclude_timestamp=False)

shows the system wall-clock time when a chat item was created. It helps with internal ordering and metrics, but it doesn’t exactly match the timing of your audio file. If you want to sync transcripts with audio playback smoothly, the best way is to enable TTS-aligned transcripts. You can do this by setting

use_tts_aligned_transcript=True

in your

AgentSession

configuration. This will give you relative timestamps like

start_time

and

end_time

for each word or phrase, perfectly aligned with the audio stream. Would you like me to walk you through the steps to enable TTS-aligned transcripts or show you how to access these timestamps in your application? I’m here to help! Sources: Cartesia TTS integration guide | LiveKit Docs | ElevenLabs TTS integration guide | LiveKit Docs | livekit-agents/livekit/agents/voice/agent_activity.py

Okay, but will this work with, for example, GBT real-time where it doesn't have a TTS, and also these timestamps - where would they be given in the transcript?

I’m glad you’re exploring GBT real-time without TTS! You can still get accurate timestamps by using the transcription features within the agent framework. Here’s a friendly walkthrough: 1. Enable a Speech-to-Text (STT) Plugin: Try adding an STT plugin like Google Cloud STT to your agent pipeline. This helps capture the timing of the user’s speech accurately. 2. Use a `transcription_node`: You can create a custom

transcription_node

in your agent to access

TimedString

objects, which provide

start_time

and

end_time

for each word or phrase—super handy! 3. Frontend Access: On the frontend side, you can subscribe to the

lk.transcription

topic to pull timing data directly from transcription events. If you’d like, I’d be happy to guide you through detailed steps for any of these options. Just let me know! Sources: Text and transcriptions | LiveKit Docs | Realtime model integrations | LiveKit Docs | Understanding Transcript Timing in Voice Conversations | examples/voice_agents/README.md | Live Conversations with AI using ChatGPT and WebRTC

I'm confused, and sorry, I meant GPT in real-time.

To get accurate timestamps for GPT in real-time without TTS, a great approach is to use the transcription features within the agent framework. Let me walk you through how you can do this: 1. Enable a Speech-to-Text (STT) Plugin: Simply add an STT plugin like Google Cloud STT to your agent pipeline. This helps capture the timing of the user’s speech. 2. Use a `transcription_node`: You can implement a custom

transcription_node

in your agent to access

TimedString

objects, which include

start_time

and

end_time

for each word or phrase—super useful! 3. Frontend Access: On the frontend side, you can subscribe to the

lk.transcription

topic to pull timing data directly from transcription events. If you’d like, I’m happy to provide more detailed steps on how to implement any of these methods. Just let me know!

I feel like these are just band-aid fixes. I just want a way to get the timing on the transcript segments of when that transcript segment started. The whole point of this is that we can map the timing to the audio. That's it. You mentioned there's a use_TTS_aligned_transcript. And I want to ask, does that add the timing to the transcript itself?

I'm glad you asked about

use_tts_aligned_transcript=True

! When you enable this, it does add timing information to the transcript, which is super helpful. However, this timing data isn’t embedded directly into a standalone transcript file. Instead, it’s delivered programmatically through the agent’s text stream and can be accessed via the

transcription_node

method. With this feature turned on, your transcription output will include precise timing details (

start_time

and

end_time

) for each word or segment, all relative to the start of the agent’s current turn. This makes it easier to align the transcript perfectly with the audio. Just a little heads-up: you’ll need to capture and process this timing data within your application logic or frontend to make the most of it. If you’d like, I’m here to help guide you through implementing this in your agent or frontend application. Just let me know! Sources: Text and transcriptions | LiveKit Docs | Cartesia TTS integration guide | LiveKit Docs | ElevenLabs TTS integration guide | LiveKit Docs

Okay, but I don't like this. Any way we can just map the transcript segments to their respective audio timings in the audio file. That's it.

I'm happy to help you understand how to map transcript segments to their audio timings! You can use the

startTime

and

endTime

fields found in the

TranscriptionSegment

interface. These timestamps show exactly when each segment begins and ends within the audio stream. If you're working on frontend applications, you can listen for the

RoomEvent.TranscriptionReceived

event using a LiveKit client SDK. This event gives you an array of

TranscriptionSegment

objects, complete with timing details for each segment. And if you're using React, the

useTrackTranscription

hook is a great way to access this information smoothly. Would you like me to walk you through detailed steps to implement this in your app, or do you need guidance tailored to a specific platform? I'm here to support you! Sources: Text and transcriptions | LiveKit Docs | lib/src/context/transcription_context.dart | transcription.go | packages/livekit-rtc/src/transcription.ts | packages/core/src/helper/transcriptions.ts