Does this implementation make sense to you or is t...
# ask-ai
c
Does this implementation make sense to you or is there a more elegant way to achieve the same? I.e. make sure we don't stream transcripts of response that the agent never actually says e.g. because it gets interrupted before.
Copy code
async def transcription_node(
        self, text: AsyncIterable[str | TimedString], model_settings: ModelSettings
    ) -> AsyncGenerator[str | TimedString, None]:
        """
        Intercepts agent transcription to stream chunks via Redis.

        IMPORTANT: To avoid orphaned streams when user interrupts before audio plays,
        we only start streaming once audio has actually started (agent_state == "speaking").
        
        This follows the same pattern as LiveKit's TranscriptSynchronizer which gates
        text output based on audio timing.

        The final message is handled separately by conversation_item_added event,
        which calls session_recorder.log_and_stream_conversation_item.
        Message ID continuity is maintained via transcription_drafts.
        """
        session_recorder = getattr(self.session.userdata, "session_recorder", None)
        
        # Track streaming state
        stream_started = False
        buffered_chunks: list[str | TimedString] = []
        
        # Small polling interval to check for audio start
        AUDIO_CHECK_INTERVAL = 0.05  # 50ms
        
        async def wait_for_audio_or_interruption() -> bool:
            """
            Wait until audio starts playing (agent_state == 'speaking') or speech is interrupted.
            Returns True if audio started, False if interrupted before audio.
            """
            while True:
                # Check if audio has started
                if self.session.agent_state == "speaking":
                    return True
                
                # Check if speech was interrupted before audio started
                speech_handle: SpeechHandle | None = getattr(self.session, "current_speech", None)
                if speech_handle and speech_handle.interrupted:
                    <http://logger.info|logger.info>("[TRANSCRIPTION] Speech interrupted before audio started - discarding buffer")
                    return False
                
                # Brief wait before checking again
                await asyncio.sleep(AUDIO_CHECK_INTERVAL)
        
        async def start_stream_and_flush_buffer():
            """Start the transcription stream and flush all buffered chunks."""
            nonlocal stream_started
            if stream_started or not session_recorder:
                return
            
            stream_started = True
            try:
                await session_recorder.start_agent_transcription_stream()
                <http://logger.info|logger.info>(
                    "[TRANSCRIPTION] Audio started - flushing %d buffered chunks",
                    len(buffered_chunks)
                )

                # Flush all buffered chunks
                for buffered_delta in buffered_chunks:
                    if buffered_delta:
                        await session_recorder.publish_agent_transcription_chunk(buffered_delta)

            except Exception as e:
                logger.warning("Failed to start transcription stream: %s", e)
        
        async for delta in text:
            # Check current speech handle for interruption
            speech_handle: SpeechHandle | None = getattr(self.session, "current_speech", None)
            
            if not stream_started:
                # Audio hasn't started yet - buffer the chunk
                if delta:
                    buffered_chunks.append(delta)
                
                # Check if audio has started (agent is now speaking)
                if self.session.agent_state == "speaking":
                    await start_stream_and_flush_buffer()
                elif speech_handle and speech_handle.interrupted:
                    # Interrupted before audio started - discard buffer and stop
                    <http://logger.info|logger.info>(
                        "[TRANSCRIPTION] Interrupted before audio - discarding %d chunks",
                        len(buffered_chunks)
                    )
                    buffered_chunks.clear()
                    # Still yield the delta to maintain the pipeline
                    yield delta
                    continue
            else:
                # Stream is active - publish chunk directly
                if session_recorder and delta:
                    try:
                        await session_recorder.publish_agent_transcription_chunk(delta)
                    except Exception as e:
                        logger.warning("Failed to publish transcription chunk: %s", e)
            
            yield delta
        
        # After all chunks processed, check if we need to start stream
        # (handles edge case where audio starts after last chunk arrives)
        if not stream_started and buffered_chunks:
            # Give a brief moment for audio to start
            audio_started = await asyncio.wait_for(
                wait_for_audio_or_interruption(),
                timeout=0.5  # Max 500ms wait
            ) if buffered_chunks else False
            
            if audio_started:
                await start_stream_and_flush_buffer()
            else:
                <http://logger.info|logger.info>(
                    "[TRANSCRIPTION] Stream ended without audio start - "
                    "discarding %d buffered chunks (no orphan created)",
                    len(buffered_chunks)
                )
t
Thanks for reaching out, Julian. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this implementation makes sense and correctly solves the main problem (preventing orphaned transcript streams when audio never actually plays). It’s clear, handles buffering, interruption, and final-edge cases well. A more elegant, lower-latency approach is to avoid polling and use event-driven signals from the speech lifecycle (start / interrupted) instead of checking
agent_state
on a timer. That simplifies the code, removes CPU/timing overhead, and eliminates certain race windows. Concise suggestions: • Use SpeechHandle lifecycle callbacks (e.g., on_start / on_interrupted) to: ◦ start the transcription stream and flush buffered chunks when audio actually starts ◦ clear the buffer when interrupted before start • If SpeechHandle events aren’t available in your SDK version, create a single small task that awaits the speech handle’s events (or state change) once per speech session — don’t poll inside the per-chunk loop. • Keep your final fallback (short wait after the last chunk) only if you expect audio to start slightly after the last text chunk; otherwise it can be removed when using event-driven handlers. • Continue to yield deltas so the rest of the pipeline stays intact; publishing to session_recorder can remain async (wrapped with try/except as you have). If you’d like, I can: • show a concise event-driven code snippet adapted to your snippet, or • sketch the small monitoring-task approach if SpeechHandle lacks event hooks. Which would you prefer? Sources: Agent speech and audio | LiveKit docs | Pipeline nodes and hooks | LiveKit docs | Agents (Python) | livekit/livekit_composite | DeepWiki | Observability data hooks | LiveKit docs | Text and transcriptions | LiveKit docs
c
I can't find the onstart callback - are you sure it exists?
t
Good catch — you’re right to question that. There is no documented `onstart`/`on_start` callback on
SpeechHandle
in the SDK sources I have. Given that, your current approach (buffering chunks and gating publish on
agent_state
/
current_speech.interrupted
) is appropriate. Two concise options moving forward: • Keep your buffered/polling approach but reduce polling surface: ◦ Poll only once per speech session (a short watcher task started when
current_speech
is created) rather than polling inside the per-chunk loop. ◦ This watcher awaits either audio-start (check
agent_state == "speaking"
) or interruption, then flushes or discards the buffer. • Use AgentSession events instead of SpeechHandle events: ◦ Listen for
agent_state_changed
(or other session-level events available in your SDK) to detect when the agent moves into a speaking state and trigger flush/start there. If you tell me which SDK (Python, Node, Flutter, etc.) you’re using, I’ll provide a compact snippet for the watcher-task approach or an example using
agent_state_changed
. Which would you prefer? Sources: client-sdk-flutter | livekit/livekit_composite | DeepWiki | Text and transcriptions | LiveKit docs | Agents and handoffs | LiveKit docs | Swift quickstart | LiveKit docs | Pipeline nodes and hooks | LiveKit docs