has anyone built something with multiple bots that can interrupt each other mid conversation based on conversation context

Thanks for reaching out, Atul. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Okay help me out with a room where 2 agents are deployed 1 is a generic help agent and other is a tech specific agent. And when a user asks a tech specific question- the generic agent starts talking by nature but I would want the tech agent to interupt the generic agent. Handle the mute and unmute carefully. Use python. Also comment on whether a bot can interrupt the user mid conversation

Thanks — this is a clear use case. I’ll provide a focused Python example that shows: • Starting two agents in the same room so they hear each other (generic help agent + tech agent). • Logic where the tech agent listens for tech-specific questions, interrupts the generic agent, and takes the floor. • Careful mute/unmute handling so only the intended agent speaks. • A short note on whether a bot can interrupt a user mid-conversation. Before the code: one quick clarifying question so I tailor the example correctly: • Are you using LiveKit’s Python SDK and the LiveKit agent/session APIs referenced earlier (so I can use

session.start

,

RoomInputOptions

,

session.interrupt()

semantics)? If yes, I’ll use those primitives. If not, tell me which SDK or platform you’re using. Example (Python) — assumes LiveKit-style session/agent APIs and that

session.interrupt()

or

session.stop()

and mute/unmute controls are available. Adapt identifiers to your SDK. 1) High-level flow • Start both agents in the same room with

participant_filter

that includes other agents so they hear each other. • Generic agent starts responding to the user by default. • Both agents transcribe incoming speech (including agent speech forwarded to inputs). • Tech agent runs a lightweight classifier or keyword/LLM check on incoming user transcript; when it detects a tech-specific question it: ◦ Asserts priority, mutes generic agent, interrupts the generic agent’s session, then un-mutes itself and begins speaking. ◦ After finishing, it can unmute the generic agent and yield the floor. 2) Example code sketch

# Pseudocode / sketch using LiveKit-like APIs in Python.
# Replace API calls with exact SDK functions in your environment.

from time import sleep
from threading import Thread
from typing import Callable

# Helpers to represent SDK behavior
class AgentSession:
    def __init__(self, name, session):
        self.name = name
        self.session = session  # actual SDK session object
        self.muted = False

    async def start(self, room, input_options=None):
        # start agent in room with input_options
        await self.session.start(agent=self.name, room=room, input_options=input_options)

    async def mute(self):
        # mute this agent's published audio
        await self.session.mute()    # SDK call to mute publishing
        self.muted = True

    async def unmute(self):
        await self.session.unmute()
        self.muted = False

    async def interrupt(self):
        # signal to stop current output (if SDK provides)
        await self.session.interrupt()

    async def speak(self, text):
        # start a TTS/agent response (SDK-specific)
        await self.session.speak(text)

    # subscribe to transcript events, agent's SDK will call on_transcript with (speaker, text)
    def on_transcript(self, speaker, text):
        pass


# User-defined simple classifier to detect tech-specific question
def is_tech_question(text: str) -> bool:
    keywords = ["install", "error", "stack trace", "refactor", "deployment", "api", "endpoint", "server"]
    text_lower = text.lower()
    return any(k in text_lower for k in keywords)


# Or use an LLM classifier for higher accuracy (call your model here)
async def classify_text_with_llm(text: str) -> bool:
    # call LLM to determine if tech-specific
    # return True if tech-specific
    pass


# Example orchestration
async def run_room(room):
    # configure both agents to hear all participants (including other agents)
    input_options = RoomInputOptions(participant_filter=lambda p: True)

    generic = AgentSession("generic_help_agent", session=GenericSDKSession(...))
    tech = AgentSession("tech_agent", session=TechSDKSession(...))

    await generic.start(room, input_options=input_options)
    await tech.start(room, input_options=input_options)

    # Hook transcript event handlers (SDK-specific wiring)
    # Assume SDK provides a way to subscribe to transcriptions with speaker id
    async def on_transcript(speaker_id, text):
        # If speaker is user, let generic normally respond (generic agent logic)
        if is_user(speaker_id):
            # Generic agent will start speaking by default. Tech agent should listen.
            if is_tech_question(text):
                # Tech agent should interrupt generic agent and take over.
                # 1. Mute generic agent's output so it won't broadcast while interrupted.
                await generic.mute()

                # 2. Interrupt generic agent's ongoing response (if SDK supports)
                await generic.interrupt()

                # 3. Unmute tech agent (ensure it can speak)
                await tech.unmute()

                # 4. Tech agent responds
                await tech.speak("I can help with that technical issue. Can you share the error message?")

                # 5. After finishing, optionally mute tech and unmute generic
                await tech.mute()
                await generic.unmute()
            else:
                # Not a tech question — generic continues as usual
                pass

        # If speaker is an agent (generic speaking), tech agent may also listen and decide to interrupt
        if is_agent(speaker_id) and speaker_id == generic.session.participant_id:
            # Evaluate the generic agent's output content; if it missed a tech cue, tech can preemptively interrupt
            if is_tech_question(text):
                await generic.mute()
                await generic.interrupt()
                await tech.unmute()
                await tech.speak("Hold on — that sounds like a technical issue; let me handle it.")
                await tech.mute()
                await generic.unmute()

    # SDK: subscribe to transcript events
    room.on("transcript", on_transcript)


# Utility predicates - depends on SDK participant object model
def is_user(participant_id) -> bool:
    # determine if the speaker is a user (vs. an agent)
    # implement according to how you label agent participants
    return not participant_id.startswith("agent_")

def is_agent(participant_id) -> bool:
    return participant_id.startswith("agent_")

3) Mute/unmute and interrupt details (practical notes) • Always mute the agent being interrupted before calling interrupt/stop to avoid overlapping audio. • Use

session.interrupt()

to stop the outgoing response if available; if not, call whatever SDK method cancels the current TTS/stream. • Ensure the interrupting agent is unmuted before starting TTS. After it finishes, mute it and unmute the previous agent to restore flow. • Add small delays (100–300 ms) between mute/unmute/interrupt to allow the room and audio pipeline to stabilize. 4) Arbitration to avoid flip-flopping • Use a priority rule: tech agent has higher priority than generic agent for tech queries. • Add cooldown: once an agent interrupts, block further interrupts for N seconds. • Require confidence thresholds from your classifier/LLM before interrupting. 5) Can a bot interrupt the user mid-conversation? • Yes, a bot can interrupt another agent in the room by muting and calling an interrupt/stop on their session and starting its own output. • Interrupting a human user is possible in technical terms if: ◦ The user is publishing audio and the system uses VAD/transcription and server-side audio control to mute/unmute participants. However, muting a human participant without consent may be a poor UX and could require explicit client support or policy decisions. • Best practices: ◦ Prefer interrupting agents rather than forcibly muting users. ◦ If you must interrupt a speaking user, use clear audible signals (chime) and keep interruptions brief. ◦ Consider asking users permission or using a visual cue before cutting audio. If you want, I can: • Provide a concrete, complete Python example using your exact LiveKit SDK bindings (give me the package names or a snippet of how you currently start agents). • Add a sample LLM classifier snippet to reduce false positives. • Draft a safe arbitration/cooldown module you can drop into your agent logic. Which should I build next for you — a full SDK-aligned code sample or an LLM-based classifier + cooldown module?

build a full sdk aligned code sample of the above also comment on: for example the user (human) is speaking wrong about computers and the tech agent wants to interrupt the user (should I keep polling llm called at every token?) how do i detect when to interrupt

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!

Okay gimme a prompt for an llm agent that summarizes our conversation along relevant context and docs you referred

@bumpy-needle-9917 you are doing the wrong approach

@billowy-hairdresser-80384 please enlighten me

Because you have to use supervisor agent and he can hand off it according to the discussion.

So i need multiple video avatars as well; and each avatar is tied to a session so I need multiple agentsessions with 1 agent and adding a coordinator/supervisor doesnt seem of much help for passing on data across the agent sessions