https://livekit.io logo
Join Slack
Powered by
I'm current ly using STT -> LLM -> TTS pipel...
# ask-ai
l
I'm current ly using STT -> LLM -> TTS pipeline for my agent. it interacting with one participant. when another participant joins the room (means take over the agent's call) so from that agent didn't geneate speech in that room. but it lisensts to the both audio and take note and genrate summary. how to do this
t
Thanks for reaching out, Rajadurai. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great question. You're asking how to keep your agent listening to all participants and creating transcripts/summaries while stopping it from speaking when a second participant joins and takes over. I’ll walk through what you can do and then ask two quick clarifying questions so I can give code tailored to your setup. What you can do (high level) • Keep the agent joined as a participant and subscribe to every remote audio track so STT runs on all participants simultaneously. That lets the agent continue transcribing both speakers even when it is not actively responding. • Stop the agent from generating or emitting TTS audio when another participant takes over by disabling its audio output or by not calling the speech-generation methods. • Collect and buffer transcriptions (from per-participant STT events) and feed them to your LLM to create a summary either periodically or at end-of-call. Concrete actions to implement this 1. Detect when another participant joins or when turn-taking changes: ◦ Listen for the room/participant events (e.g., 
participant_joined
, 
user_state_changed
, or similar turn-taking events). 1. Switch the agent into a passive/listening mode: ◦ Disable audio output: call the session/output API to turn off agent audio (for example, 
session.output.set_audio_enabled(False)
), or stop calling 
session.say()
/ TTS generation routines. ◦ Keep STT active on all subscribed tracks so transcription continues. 1. Buffer transcriptions: ◦ Append each 
user_input_transcribed
(or equivalent) event to a transcript buffer along with participant identity and timestamps. 1. Generate summary with the LLM: ◦ When you want the summary (on demand, periodically, or at call end), pass the buffered transcript to the LLM with a summarization prompt and store/emit the result. Why this works • The agent remains in the room and subscribed to audio tracks, so STT continues for every participant. • Disabling audio output or avoiding TTS prevents the agent from interrupting or speaking over the live participants. • The LLM can produce summaries from the buffered transcripts independent of whether the agent speaks. Quick clarifying questions 1. Are you using Python or Node.js for your agent code? 2. Do you want the agent to stop speaking immediately when any second participant joins, or only when that participant is actively speaking/taking the turn? 3. Do you want summaries in real time (e.g., every N seconds/minutes) or only at the end of the call? If you tell me your language (Python/Node) and which summary timing you prefer, I’ll provide a short example snippet showing: • listening for the participant join/turn event, • disabling audio output while keeping STT active, • collecting transcripts and invoking the LLM to summarize. Would you like a Python example or a Node.js example next? Sources: livekit/livekit_composite | DeepWiki | Agents Framework | livekit/livekit_composite | DeepWiki | Text and transcriptions | LiveKit Documentation | Building multi-agent architectures with LiveKit agents | LiveKit | Agent Builder | LiveKit Documentation
l
1.I'm using python 2. I want to stop the agent when the participant is taking the turn 3.I need summary at end of the call
t
Thanks — this is clear and helpful. Below is a concise, practical Python plan and example you can use to: • Stop the agent from speaking whenever another participant takes the turn, • Continue transcribing all participants the whole time, and • Produce a final summary at the end of the call. I’ll assume you’re using LiveKit’s agent APIs and a typical STT/LLM/TTS plugin setup. High-level flow • Join the room as the agent and subscribe to all remote audio tracks so STT runs for every participant. • Monitor participant speaking state (turn-taking) via 
user_state_changed
or similar event. • When a non-agent participant becomes the active speaker, disable the agent’s audio output so it won’t generate TTS, but keep STT/transcription active. • Buffer all transcriptions (with participant identity and timestamps). • On session end (or when the room empties), send the buffered transcript to the LLM to generate a final summary. Key behaviors to implement • Disable speech output: use 
session.output.set_audio_enabled(False)
(or the equivalent in your session object) when another participant takes the turn. • Keep transcription: do not disable text output; continue handling 
user_input_transcribed
(or 
conversation_item_added
) events. • End-of-call summary: trigger LLM summary in the shutdown hook or when the room becomes empty. Minimal Python example (Adapt to your project structure and the exact LiveKit SDK objects you use.)
Copy code
import asyncio
from typing import List
from livekit.agents import JobContext, AgentSession
from livekit import rtc

async def entrypoint(ctx: JobContext):
    # Create agent session with your STT/LLM/TTS plugins
    session = AgentSession(
        room=ctx.room,
        # stt=..., llm=..., tts=..., vad=...
    )

    transcript_buffer: List[str] = []
    agent_identity = ctx.room.local_participant.identity  # agent's identity

    # Collect every transcription (keeps transcribing both participants)
    @session.on("user_input_transcribed")
    def on_transcribed(event):
        # event.participant_identity, event.transcript (adjust according to actual event shape)
        transcript_buffer.append(f"{event.participant_identity}: {event.transcript}")

    # Track speaking/turn-taking and mute agent audio when a non-agent participant speaks
    @session.on("user_state_changed")
    def on_user_state_changed(event):
        # event.participant_identity and event.new_state (e.g., "speaking" or "silent")
        if event.participant_identity != agent_identity and event.new_state == "speaking":
            # Another participant is taking the turn — stop agent TTS output
            session.output.set_audio_enabled(False)
        elif event.participant_identity != agent_identity and event.new_state == "silent":
            # Optionally re-enable agent output when they finish
            # Only re-enable if you want agent to speak again when the remote stops
            session.output.set_audio_enabled(True)

    # Optionally detect participant join to prepare for turns
    @ctx.room.on("participant_joined")
    def on_participant_joined(participant: rtc.RemoteParticipant):
        # You may want to log or track participants here
        print(f"Participant joined: {participant.identity}")

    # Start the session (this will run until closed)
    await session.start()

    # Wait until session ends (depends on how your app ends sessions)
    await session.wait_closed()

    # On session close — generate final summary
    if transcript_buffer:
        summary_prompt = "Summarize the following conversation concisely:\n\n" + "\n".join(transcript_buffer)
        # Call your LLM plugin to generate summary; adjust to plugin API
        llm_response = await session.llm.chat.create(messages=[{"role":"user","content": summary_prompt}], max_tokens=250)
        summary = llm_response.choices[0].message.content
        print("Call summary:\n", summary)
        # Save/send summary as needed
Practical notes and tips • Use VAD and 
user_state_changed
events for reliable turn detection instead of only participant joins. That lets you mute the agent only when the other participant is actively speaking. • If you want the agent to remain able to speak again after the other stops, re-enable audio output when their state returns to idle/silent. • Buffer format: include participant identity and timestamps for clearer summaries and context. • Trigger summary when the session ends or when the room becomes empty (e.g., detect 
participant_left
and check participant count). • If you don’t want the agent to ever speak once a second participant appears, you can permanently keep 
audio_enabled
false until you explicitly decide to re-enable it. Would you like: • A fully fleshed-out runnable sample tailored to a specific STT/LLM/TTS plugin (e.g., Deepgram + OpenAI + your TTS), or • A version that keeps the agent muted only while the other participant is actively speaking (so the agent can speak again between turns)? Tell me which option and I’ll produce the code snippet specific to your plugins. Sources: Pipeline nodes and hooks | LiveKit Documentation | Turns overview | LiveKit Documentation | How to implement call greeting retries in voice agents | LiveKit | Building multi-agent architectures with LiveKit agents | LiveKit | How to build an agent with speech-to-text input and text-only output | Live...
2 Views
Open in Slack
PreviousNext
Powered by