https://livekit.io logo
Join Slack
Powered by
Context: We’re building real-time captions that ap...
# ask-ai
l
Context: We’re building real-time captions that appear line-by-line as the agent speaks. For text-based models (Haiku + Cartesia TTS), we use 
SpeechHandle._wait_for_generation()
to detect when text generation completes — this lets us show the last caption line ~13ms after the LLM finishes, while TTS audio is still playing. Problem: For Realtime models (GPT-4o Realtime), 
_wait_for_generation()
fails immediately with 
"cannot use wait_for_generation: no active generation is running"
because Realtime generates audio and text simultaneously — there’s no separate “text generation” phase. 
agentState
stays 
speaking
until audio playback finishes (8-12s after text stops growing). We currently fall back to a 3-second inactivity timer to detect that text has stopped, which adds noticeable delay. Question: Is there an SDK event, hook, or property on 
SpeechHandle
/ 
AgentSession
that fires when a Realtime model’s text transcript is complete (i.e., the model has finished its response), while audio buffers are still draining? We’re looking for the equivalent of 
_wait_for_generation()
but for the Realtime path — the earliest possible signal that no more text/transcript content will arrive for this turn.
✅ 1
t
Thanks for reaching out, Tom. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about whether there’s an SDK event/hook that signals Realtime model text completion (the equivalent of 
_wait_for_generation()
for the Realtime path). short answer There is no SDK event or property that reliably signals “text transcript complete” for Realtime models while audio is still draining. 
_wait_for_generation()
only applies to the cascaded/text-generation path and fails for Realtime models because text and audio are produced together in a single generation. What that means for you • Realtime agents (e.g., GPT-4o Realtime) do not expose a separate “text finished” phase you can wait on; agentState remains 
speaking
until audio playout completes. • Your 3-second inactivity timer is the practical workaround today, but it does add latency. Alternatives to consider • Check Realtime model metrics: some Realtime implementations emit post-response metrics (`metrics_collected`/RealtimeModelMetrics) that include timing info you might use heuristically to detect response end. (This is not a guaranteed text-complete event, but can reduce false positives.) • Use TTS-aligned transcripts or compatible TTS providers when possible: if you can switch to a TTS path that supports 
use_tts_aligned_transcript=True
, word-level timings can indicate when text output finished. This isn’t available for Realtime models that do TTS internally. • Transcription node / TimedString: route transcripts through a transcription node or use TimedString-style transcripts to infer finality; these are experimental and may not be precise. • Hybrid approach: keep your short inactivity timer but shorten it using heuristics (e.g., detect stable end-of-sentence punctuation, token-rate slowdown, or metrics-derived timing) to cut latency without false positives. If you want, tell me: • Which Realtime model/provider you’re using (GPT-4o Realtime, OpenAI Realtime, etc.), and • Whether you can switch TTS providers or the pipeline (cascaded vs Realtime) I can then suggest the most practical heuristic or metric to try for lower-latency captions. Sources: Realtime models overview | LiveKit Documentation | How to detect when an agent has finished speaking | LiveKit | Realtime model metrics | LiveKit Documentation | Text and transcriptions | LiveKit Documentation, with interactive elements
l
We’re using Realtime 1.5
t
Thanks — you're using Realtime 1.5. I’m responding to your question about whether there’s an SDK event/hook that signals when a Realtime model’s text transcript is complete (the equivalent of 
_wait_for_generation()
). Short answer Realtime 1.5 does not provide a reliable SDK event/property that signals “text transcript complete” while audio is still draining. 
_wait_for_generation()
only works for the cascaded/text-generation path and fails for Realtime because text and audio are produced together. What that means • Agent state stays 
speaking
until audio playout finishes, so there’s no separate text-complete phase you can wait on in the SDK. • The 3s inactivity timer is the practical fallback today; Realtime 1.5 doesn’t change this fundamental limitation. Practical alternatives to reduce latency • Use Realtime model metrics (if emitted) — `metrics_collected`/RealtimeModelMetrics can include timing you may use heuristically to infer response end. • Shorten your inactivity timer with heuristics: end-of-sentence punctuation, token-rate slowdown, or detecting no transcript growth for a shorter, dynamic window. • Run a separate lightweight STT/VAD alongside Realtime (or enable TTS-aligned transcripts when using a compatible TTS path) to get earlier transcript-final signals—this requires architectural change. • Route transcripts through a `transcription_node`/TimedString handler and treat partials as captions (trade accuracy for latency). Next step If you want, tell me which heuristics you’d prefer (metrics-based vs punctuation/token-rate vs external VAD+STT) and I’ll outline a concise implementation plan for that approach. Sources: Realtime models overview | LiveKit Documentation | Understanding Transcript Timing in Voice Conversations | examples/voice_agents | livekit/livekit_composite | DeepWiki | Streamline troubleshooting with Agent Observability | LiveKit | Pipeline nodes and hooks | LiveKit Documentation, with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by