Thanks — this is a clear description and a good place to start. I’ll address each of your questions and give a practical path forward. Acknowledgement • You want the client-side “seconds actually heard” (speaker output) to reveal Cartesia word timestamps accurately. • Your current wall-clock + Date.now() approach drifts during playback stalls, and HTMLAudioElement.currentTime is unreliable for WebRTC streams. Short answers • There is no LiveKit client API that directly reports “seconds actually emitted to the speaker.” • Your idea of using RTCInboundRtpStreamStats.concealedSamples to compute cumulative stutter and subtract it from wall-clock elapsed time is a sound approach. • You should also account for playout speed adjustments like `insertedSamplesForDeceleration`/`removedSamplesForAcceleration` if they appear in stats, because they change delivered audio duration. • There is no server-side playout schedule or built-in server message that reports exact client playout timing; server-side helpers like

SpeechHandle.wait_for_playout()

exist internally but their timing is not exposed to clients by default. Practical implementation guidance 1. Compute an adjusted playout position: ◦ Track when the agent starts producing audio (your

agentState

→ 'speaking' timestamp). ◦ Periodically call

RemoteAudioTrack.getRTCStatsReport()

and read inbound audio stats. ◦ Convert samples to seconds using the sample rate: ▪︎ stutterSeconds = delta(concealedSamples) / sampleRate ▪︎ decelSeconds = delta(insertedSamplesForDeceleration) / sampleRate (subtract) ▪︎ accelSeconds = delta(removedSamplesForAcceleration) / sampleRate (add) ◦ AdjustedPlayout = wallClockElapsed - cumulativeStutterSeconds - cumulativeDecelSeconds + cumulativeAccelSeconds 1. Polling cadence and deltas: ◦ Poll stats at a modest interval (e.g., 250–500 ms). Use deltas between successive reports to compute changes. ◦ Use cumulative counters (stats are monotonic) to avoid missed samples if a poll is delayed. 1. Sample rate and mapping: ◦ Use the actual codec/sampleRate from the inbound track stats (or know your TTS sample rate) for accurate conversion. ◦ Align TTS word timestamps to the same timebase as your start-of-playout event. 1. Edge cases and gotchas: ◦

concealedSamples

covers loss/underrun concealment, but some platforms may implement concealment differently; test on target browsers/devices. ◦ Jitter buffer increases can delay first-sample playout after start; initial offset should be measured per session. ◦ If users change volume or route audio to Bluetooth/headset, that doesn’t affect these counters but can change perceived latency—keep UX tolerant to small offsets. ◦ If you see

insertedSamplesForDeceleration

/

removedSamplesForAcceleration

in your stats, include them as above to correct for clock compensation. ◦ HTMLAudioElement.currentTime is not reliable for WebRTC MediaStream playout under jitter — don’t use it for word-level sync. 1. If you need absolute confirmation from the agent/server: ◦ Because there’s no built-in server-side event that tells the client “frame X has been emitted to speakers,” implement a custom signal: ▪︎ Have the agent send data messages at key points (e.g., on playout start, on TTS chunk boundaries, or on playout complete) from the server/agent when it calls its internal playout-complete hooks. ▪︎ Combine server-sent playout events with client-side RTC stats to get robust sync (server timestamps give authoritative schedule; client stats correct for network/runtime deviations). Next step I can help with • If you’d like, I can provide a concise code sketch showing: ◦ how to poll

getRTCStatsReport()

and compute the adjusted playout time, or ◦ an example data-message schema for agent->client playout events to pair with the client-side corrections. Which would you prefer? Sources: client-sdk-js | livekit/livekit_composite | DeepWiki | Why WebRTC beats WebSockets for realtime voice AI | LiveKit | How to detect when an agent has finished speaking | LiveKit, with interactive elements