Context: We have a realtime model (OpenAI GPT-4o...
# ask-ai
b
Context: We have a realtime model (OpenAI GPT-4o Realtime via Azure). Inside a @function_tool, we call generate_reply() to deliver a goodbye message, then hangup(). The goodbye speech (S1) gets interrupted by user VAD. Problem we discovered: The SDK cancels any new response (S2) while the function_tool coroutine is still executing (we confirmed via response done but not complete with status: cancelled in the same tick as the tool completing). So we can't wait for S2 inside the tool — we must return first. What we're doing now: async def hangup(self, speech): await speech.wait_for_playout() if speech.interrupted: # Return from tool immediately — releases tool context so SDK can generate S2 self._teardown_task = asyncio.create_task(self._wait_for_reply_and_teardown(speech)) else: await self._do_teardown() async def _wait_for_reply_and_teardown(self, prev_speech): # Poll session.current_speech every 50ms until S2 appears for _ in range(20): # 1s window — but S2 takes ~1s to appear, too tight await asyncio.sleep(0.05) candidate = self.session.current_speech if candidate is not None and candidate is not prev_speech: await asyncio.wait_for(candidate.wait_for_playout(), timeout=5.0) break await delete_room() Questions: 1. Is there an event or awaitable to know when the next speech handle has been scheduled after a tool returns — instead of polling session.current_speech? (e.g. session.once("speech_created", ...) or similar) 2. What is the correct pattern for "play goodbye → if interrupted → let agent respond → then hang up"? 3. Is there a way to await the next agent turn completing, rather than watching current_speech? 4. Does AgentSession expose any wait_for_next_speech() style API?
t
Thanks for reaching out, Aleh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great summary of the problem and your current workaround. I’ll respond to each of your numbered questions and then give a concise recommended pattern you can implement instead of polling. Acknowledgement: you’re asking how to detect when the agent schedules the next speech after your tool returns (so you can await the new turn and then delete the room), and how to avoid polling while still supporting the “play goodbye → if interrupted → let agent respond → then hang up” flow. Answers 1) Is there an event or awaitable to know when the next speech handle has been scheduled after a tool returns? • Yes. AgentSession emits a speech_created event whenever a new speech is initiated. You can attach a one‑time listener (for example
session.once("speech_created", callback)
) to be notified when the next speech is created instead of polling
session.current_speech
. 2) What is the correct pattern for "play goodbye → if interrupted → let agent respond → then hang up"? • Return from the tool immediately after initiating the goodbye speech. If the goodbye is interrupted, register a one‑time
speech_created
listener. When
speech_created
fires, await the new speech handle’s playout completion using
candidate.wait_for_playout()
and then run your teardown (e.g.,
delete_room()
). This ensures the SDK can schedule the next response (S2) after the tool returns and you wait for the agent turn to finish before hanging up. 3) Is there a way to await the next agent turn completing, rather than watching current_speech? • There isn’t a single “wait_for_next_turn” helper, but the pattern above (listen for
speech_created
→ get the new speech handle → await its
wait_for_playout()
) is the correct way to await the agent’s next turn without polling. 4) Does AgentSession expose any wait_for_next_speech() style API? • No. AgentSession does not expose a
wait_for_next_speech()
API. Use the
speech_created
event and then await the returned speech handle’s
wait_for_playout()
. Recommended implementation sketch 1. When you play the goodbye speech from inside the tool, return from the tool immediately (do not block waiting for S2). 2. If the goodbye speech was interrupted: ◦ Use
session.once("speech_created", callback)
to catch the next speech. ◦ In the callback, obtain the new speech handle (the callback should receive it or you can read
session.current_speech
) and await
candidate.wait_for_playout()
with a reasonable timeout. ◦ After awaiting the new speech playout completion, run your teardown (delete room etc.). 1. If goodbye was not interrupted, carry out teardown immediately. Notes and timing tips • Give a slightly larger window for the next speech to appear (your 1s polling window was tight). The
speech_created
event removes the need to poll and avoids timing races. • Use a timeout on
wait_for_playout()
(e.g., 5–10s) to avoid hanging if the agent’s speech never appears or stalls. If you’d like, I can: • Show a short async code snippet using
session.once("speech_created", ...)
and
candidate.wait_for_playout()
. • Help adapt this pattern into your existing
hangup()
/
_wait_for_reply_and_teardown()
flow. Which would you prefer? Sources: Agent speech and audio | LiveKit Documentation | Agent session | LiveKit Documentation | Agents telephony integration | LiveKit Documentation | How to detect when an agent has finished speaking | LiveKit