Costream STT Agent Architecture — Problem, Attempts, and Proposed Direction Background Costream is a live-streaming production platform built on LiveKit. We’re building a real-time speech-to-text (STT) pipeline that captures live transcripts for two purposes: 1. Live transcript UI — admin can see what’s being said in real-time 2. “What just happened” AI feature — an AI reads the transcript + play-by-play data to produce real-time summaries (e.g., sports broadcasting) The transcript is written as a WebVTT file to Cloudflare R2 on each segment batch, and the frontend polls the R2 URL. Room Architecture Costream uses a multi-room LiveKit architecture: - Event room (event_{eventId}) — where external sources publish (e.g., an NFL broadcast via RTMP/WHIP/SRT). Contains tracks like event_{id}:main_landscape. - Costream room (costream_{userId}_{eventId}) — where the host and crew are. Contains crew webcam/mic tracks (forwarded from individual costreamer rooms) and the host’s own tracks. The compositor also connects here for the crew layer, and separately to the event room for source video. A single event may have one event room and multiple costream rooms (one per crew leader). The broadcast audio (sportscasters, game commentary) is in the event room. Crew commentary is in the costream rooms. What We Built (COS-389, Phase 1) A LiveKit agent using the livekit-agents SDK (v1.5.1) with AgentSession + Agent subclass pattern (based on the official multi-user-transcriber.py example): - Transcriber(Agent) subclass with Deepgram STT (nova-3-general) - MultiUserTranscriber manages per-participant AgentSession instances - SegmentBuffer batches segments and posts to our backend every 5s or 10 segments - Agent dispatched via agentDispatchClient.createDispatch() when admin enables transcription Problems Encountered 1. Multiple AgentSession handler conflict When the agent connects to a room with multiple participants (e.g., a costreamer + an egress participant + an event source participant), MultiUserTranscriber creates an AgentSession per participant. The second session fails with: Failed to start session for costreamer_xyz: byte stream handler for topic ‘lk.agent.session’ already set This happens even with text_input=False and participant_identity set in RoomOptions. The lk.agent.session byte stream handler appears to be registered at the Room level by the first AgentSession, and subsequent sessions on the same room can’t register it again. Result: Only the first participant gets transcribed. If an egress participant connects first, it claims the handler and the actual costreamer’s session fails. 2. AgentSession is designed for conversational AI, not passive transcription We’re fighting the abstraction at every turn: - We override on_user_turn_completed just to suppress LLM response generation (raise StopResponse()) - The turn-taking / interruption detection logic is unnecessary for passive transcription - We can’t easily get Deepgram’s word-level timestamps through the Agent.on_user_turn_completed callback (it only provides text) - To get timestamps, we had to override stt_node to intercept SpeechEvent objects before they enter the agent pipeline 3. Timestamp accuracy The original agent used rough elapsed-time estimates (time.time() - start_time) for segment timestamps. We need accurate wall-clock timestamps for the “what just happened” AI feature (which needs to correlate transcript with play-by-play data at specific times). Deepgram provides accurate start_time/end_time relative to audio stream start, but these aren’t accessible through the AgentSession → on_user_turn_completed path. What We’re Proposing Drop AgentSession/Agent entirely and use the Deepgram STT plugin directly. The agent would: 1. Keep WorkerOptions + cli.run_app for LiveKit agent registration and dispatch 2. In the entrypoint, connect to the room via JobContext with AutoSubscribe.AUDIO_ONLY 3. Listen for track_subscribed events on the room 4. For each audio track, create a deepgram.STT(model=“nova-3-general”).stream() instance 5. Push rtc.AudioFrame data into the stream via stream.push_frame(frame) 6. Consume SpeechEvent objects from the async iterator, which contain SpeechData with Deepgram’s accurate start_time/end_time 7. Record wall-clock anchor time when each stream starts; compute absolute timestamps as anchor + deepgram_offset 8. Post segments to SegmentBuffer with accurate timestamps, tagged with participant identity Why this approach: - No handler conflicts — no AgentSession means no lk.agent.session byte stream registration - Multi-track support — one STT stream per audio track, independently managed - Accurate timestamps — direct access to Deepgram’s SpeechData.start_time/end_time - Simpler code — removes the conversational AI abstractions (LLM, TTS, turn-taking, interruption detection) we don’t use - Per-track attribution — each transcript segment tagged with the source participant identity - Not reinventing the wheel — still uses livekit-agents for dispatch/lifecycle and livekit-plugins-deepgram for STT; just skips the AgentSession orchestration layer Deployment model: - One agent dispatched per room (not per participant) - Event room agent → captures broadcast audio (shared across all costreamers for the event) - Costream room agent → captures crew commentary - This means the NFL broadcast audio is transcribed once in the event room, not N times per costream room Questions for Reviewers 1. Is bypassing AgentSession and using deepgram.STT().stream() directly a supported/recommended pattern for passive transcription agents? Are there lifecycle concerns we’d miss (reconnection, cleanup)? 2. Is there a better way to handle the lk.agent.session byte stream handler conflict when running multiple AgentSession instances on the same room? 3. For the track_subscribed → push_frame pattern, is there an existing helper in the SDK for piping an rtc.AudioTrack into an stt.SpeechStream, or do we need to set up the frame callback ourselves? 4. Any concerns about running separate STT streams per track from a resource/cost perspective?

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!