I am trying to implement karaoke-style word highlighting using tts word timestamps. is there a way to know how much audio has played on the client side so that I know which word to highlight at any given moment while the agent is speaking?

Thanks for reaching out, jisangyou1. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

i am using react native. i dont want live transcriptions (the user should be able to see the full response text while the agent is speaking it. we should highlight the currently spoken word).

Thanks — you're using React Native and want the full response text visible while highlighting the currently spoken word (no live streaming transcripts). Below is a focused plan and a small, actionable implementation approach. Acknowledgement • You’ll receive TTS word timestamps from the agent (enable

use_tts_aligned_transcript=True

and

sync_transcription=True

on the AgentSession/TextOutputOptions) and keep the full response text in the UI while highlighting words as audio plays. Recommended approach (React Native) 1. Receive the response: ◦ Get the full response text plus per-word timestamps as

TimedString

objects (

start_time

/

end_time

in seconds). 1. Play audio and track playback time: ◦ Use a React Native audio player that exposes playback position: ▪︎ Example libraries:

expo-av

(Audio.Sound,

getStatusAsync().positionMillis

) or

react-native-video

(

onProgress

with

currentTime

). ◦ Start playback and poll or subscribe to the player’s time updates (onProgress or setInterval reading

positionMillis

) to get current playback time in seconds. 1. Map playback time → highlight: ◦ Keep the words and their timestamps in an array [{text, start_time, end_time}, ...]. ◦ On each time update compute currentTime = playerPositionSeconds and find the active word:

word.start_time <= currentTime && currentTime <= word.end_time

. ◦ Update UI to highlight that word; render full text but style the active word differently. 1. Handle edge cases: ◦ On pause/seek/interruption: read the player’s currentTime and recompute active word. ◦ If audio starts slightly delayed vs timestamp base, record playbackStartClock (Date.now()) when you call play and use either player.currentTime or (Date.now()-playbackStartClock) to compute position; prefer player’s reported position when available. ◦ If the agent speech can be interrupted, listen for agent playback events (

playback_finished

or

agent_state_changed

) to clear highlighting. Minimal code sketch (conceptual) • Use

expo-av

example flow:

// load audio
const sound = new Audio.Sound();
await sound.loadAsync({ uri: audioUrl });
// start playback
await sound.playAsync();
// poll position
const interval = setInterval(async () => {
  const status = await sound.getStatusAsync();
  const currentTime = status.positionMillis / 1000;
  // find active word by timestamps and set state for UI highlight
}, 50);

• Render full response text as words (split by tokens) and apply highlight style to the active index from state. If you want, I can provide: • A complete React Native +

expo-av

code example that converts LiveKit

TimedString

objects into highlighted text, • Or a

react-native-video

version with

onProgress

usage. Which one would you like? , with interactive elements

this is for during a live voice call

Thanks — confirming: you want karaoke-style highlighting during a live voice call in React Native, showing the full response text while highlighting the currently spoken word (no streaming transcript). That’s doable; here’s a concise plan and the question I need to give a concrete code sample. What to do (summary) 1. Get per-word timestamps from the agent: ◦ Enable

use_tts_aligned_transcript=True

and

sync_transcription=True

so you receive

TimedString

objects with `start_time`/`end_time` for the whole utterance. 1. Detect when agent audio starts in the call: ◦ Use LiveKit’s agent playback events (

agent_state_changed

to

speaking

or

playback_finished

) or detect the remote audio track becoming audible on the client. When audio becomes audible, record a playback start time (see options below). 1. Use the client audio clock as authoritative: ◦ In the live call, use the actual audio playback position reported by your RN audio player (preferred) or compute elapsed time from the recorded playback start time (fallback). ◦ Map playback time (seconds) to the word timestamps and highlight the word where

start_time <= playbackTime <= end_time

. 1. Handle interruptions/seeks: ◦ On pause/interrupt/resume or when agent playback ends, read the player position and recompute the active word; clear highlights when finished. Client-side options (React Native) • If you use a player that exposes position updates: ◦ expo-av:

Audio.Sound.getStatusAsync().positionMillis

polled via interval or

setOnPlaybackStatusUpdate

. ◦ react-native-track-player:

TrackPlayer.getPosition()

or

progress

events. ◦ react-native-video:

onProgress

with

currentTime

. • If you can’t get a reliable position, record Date.now() when the remote audio becomes audible and compute elapsed = (Date.now() - startTimestamp)/1000. Timing tips • Prefer the player’s reported

currentTime

or AudioContext clock for accuracy. • Use small polling interval (e.g., 30–100ms) or the player’s callback to update highlight. • Use the transcript timestamps only as the mapping reference; rely on the audio clock for real playback timing. • If audio start has non-zero latency, align by recording the exact moment the user hears audio (or use the player’s first-play event). Next step Which audio playback component are you using in React Native (expo-av, react-native-track-player, react-native-video, or something else)? I’ll provide a concise, runnable example for that library showing how to load remote agent audio, poll position, and highlight words using the

TimedString

timestamps. , with interactive elements