https://livekit.io logo
Join Slack
Powered by
I have some experience with how agents behave in a...
# ask-ai
f
I have some experience with how agents behave in a room with only one participant, I think agent gets attached to a given user and then starts listening to it.. I have deployed agent worker process which attaches to the user and responds to user.. I want to know and understand how agents work with multiple user personas in a room.. How to make it listen to every person and identify everyone uniquely.. Sample usecase is creating an agent who can listen to a group conversation and then give suggestion etc... Just tell how these agents work in concept under lk.
t
Thanks for reaching out, Tushar. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great question. I’ll explain how LiveKit agents behave conceptually in a multi-user room, how they can listen to every person, and how you can uniquely identify and process each speaker so the agent can give group-aware suggestions. Acknowledging your goal • You want an agent that listens to a group conversation, identifies each speaker uniquely, and responds or suggests things based on the multi-person context. How agents appear in a room • Agents join a LiveKit room as participants of kind 
AGENT
. They act like participants: they can receive audio from other participants and publish their own audio or data messages back to the room. • By default, agents receive audio from Standard Participants and SIP callers, but not from other agents unless configured otherwise. Listening to multiple participants • The agent receives separate audio tracks published by each participant. The platform routes each participant’s audio to the agent worker so the agent can process them concurrently. • You can explicitly control which participant kinds the agent listens to using 
participant_kinds
in RoomOptions. Include 
AGENT
if you want agents to hear other agents as well. Uniquely identifying who is speaking • Every participant has a unique 
identity
string. Use that 
identity
to tag transcriptions, maintain per-user state, or route messages. • When tracks are subscribed, events include the RemoteParticipant object (or equivalent) so you can access 
participant.identity
and associate audio/transcripts with that identity. Handling interleaved/group audio and speaker activity • Audio from different users can interleave. Use voice activity detection (VAD) and per-track processing so you know which track (hence which participant identity) is currently active. • Transcription pipelines will emit utterances with timestamps and speaker association when you process per-participant tracks. Keep a rolling, timestamped conversation buffer with speaker tags to reconstruct conversation flow. Design patterns for group-aware behavior • Conversation buffer: store recent utterances with speaker identity and timestamps. This buffer is what you feed to your LLM or reasoning layer so it can produce context-aware suggestions. • Triggering: decide how the agent decides to act — e.g., on silence (end of turn), a keyword, periodic summaries, or an explicit user command. • Context windowing: include recent turns and speaker identities so the agent can, for example, say “Alice, you suggested X earlier…” rather than a generic reply. How the agent responds to the group • By default the agent publishes an audio track into the room — everyone in the room hears the agent’s spoken response. • If you need per-user responses (private replies), send targeted data messages to specific participants instead of (or in addition to) broadcasting audio. You can also implement logic to create one-to-one audio channels, or alternatively synthesize audio and send it privately via data or external channels. Focusing on a single participant in a multi-user room • If you want the agent to focus on a single participant at times, use “fixed participant” mode (RoomIO or session APIs that let you set a 
participant
identity). This makes the agent process only that participant’s audio stream while still in the shared room. Lifecycle and dynamic behavior • The agent receives participant connected/disconnected events, so it can adapt in real time: update participant lists, change listening targets, or change behavior based on who’s present. • The agent job typically runs while participants remain or until the agent is explicitly disconnected. Practical example of a group-suggestion agent (conceptual steps) 1. Configure the agent’s RoomOptions to include the participant kinds you want it to listen to. 2. Subscribe to per-participant audio tracks and run VAD + STT per track. 3. Tag each transcript with the participant 
identity
and timestamp; append to a rolling conversation buffer. 4. Trigger an LLM prompt when appropriate (e.g., end of discussion, a pause, or a request). Include recent turns and speaker tags in the prompt. 5. Publish the agent’s response as an audio track to the room for everyone, or send targeted data messages for private suggestions. When to choose broadcast audio vs. targeted responses • Broadcast audio (agent speaks into room): best for shared suggestions, summaries, and facilitation. • Targeted data messages or private audio: best for individualized advice or confidential suggestions. If you want, I can next: • Show a short code sketch that demonstrates subscribing to per-participant tracks and tagging transcripts with identities, or • Walk through how to structure the LLM prompt with speaker tags and timestamps for better context understanding. Which would help you more: a code sketch for track subscription & STT wiring, or an example LLM prompt/template that includes speaker identity and timestamps? Sources: sip | livekit/livekit_composite | DeepWiki | Agents Framework | livekit/livekit_composite | DeepWiki | Agent session | LiveKit Documentation | How to Test Your Agent Using Another Agent | LiveKit | Job lifecycle | LiveKit Documentation
f
participant_identity: NotGivenOr[str] = NOT_GIVEN """The participant to link to. If not provided, link to the first participant. Can be overridden by the 
participant
argument of RoomIO constructor or 
set_participant
.""" Can you give me the reference of how to start agent session with agent linked to all participant by default
t
Thanks — I see you’re asking for a reference and example for starting an agent session that’s “linked to all participants by default,” and you pointed to the 
participant_identity
behavior. Short answer • LiveKit Agents do not support being linked to all participants by default. An agent is linked to a single participant at a time. If 
participant_identity
is not provided, the agent links to the first participant that joins the room. What 
participant_identity
does • 
participant_identity: NotGivenOr[str] = NOT_GIVEN
means: if you don’t pass a value, the agent links to the first participant. You can pass a specific identity to link the agent to that particular participant instead. Example: start session that links to the first participant (default)
Copy code
await session.start(
    agent=my_agent,
    room=ctx.room,
    room_options=room_io.RoomOptions(
        participant_identity=None  # defaults to first participant
    )
)
Options to handle multiple participants • Programmatically switch the linked participant during the session: ◦ Use 
RoomIO.set_participant("participant-identity")
(or equivalent) to change which single participant the agent is linked to at runtime. • Manually subscribe to multiple participant tracks: ◦ Instead of relying on the single linked-participant flow, subscribe to per-participant audio tracks (JobContext / track subscription) and run STT/VAD per track. Tag transcripts with 
participant.identity
so the agent can reason about everyone simultaneously. • Broadcast vs. private responses: ◦ The agent’s audio is broadcast to the room by default. For private replies, send targeted data messages to specific participants. References • Agent sessions link to a single participant and default to the first participant when none is provided: https://docs.livekit.io/agents/logic/sessions/ • Job/track subscription patterns for more advanced multi-participant handling: https://docs.livekit.io/agents/server/job/ If you’d like, I can: • Show a short code sketch demonstrating subscribing to multiple participant audio tracks and tagging transcripts with 
participant.identity
, or • Show how to call 
set_participant()
to rotate focus between participants. Which one would help you more? Sources: Agent session | LiveKit Documentation | Job lifecycle | LiveKit Documentation
f
Show a short code sketch demonstrating subscribing to multiple participant audio tracks and tagging transcripts with 
participant.identity
, or
t
Below is a short Python code sketch that demonstrates the conceptual flow for subscribing to multiple participants’ audio tracks in a LiveKit agent job, running per-track STT, and tagging each transcription with 
participant.identity
. This sketch focuses on structure and key calls — adapt the actual STT and audio handling to your libraries and runtime.
Copy code
# Conceptual sketch — adapt imports and runtime to your project
from livekit import rtc  # hypothetical import names for clarity
from livekit.agents import JobContext, RemoteParticipant, RemoteAudioTrack

async def handle_track(track: RemoteAudioTrack, participant: RemoteParticipant, job_ctx: JobContext):
    """
    Per-track processing: read audio frames, run VAD/STT, emit transcripts tagged with participant.identity
    """
    # Example: stream audio frames from the remote track
    async for audio_frame in track.iter_frames():
        # 1) Run VAD to detect speech segments (pseudo)
        if not is_speech_frame(audio_frame):
            continue

        # 2) Buffer frames until end-of-utterance, then send to STT
        utterance_bytes = collect_until_end_of_utterance(track)

        # 3) Call your STT (synchronous or async). Replace with your STT client call.
        transcript_text = await my_stt_client.transcribe(utterance_bytes)

        # 4) Tag transcript with participant.identity and timestamp
        transcript = {
            "identity": participant.identity,
            "timestamp": current_timestamp(),
            "text": transcript_text
        }

        # 5) Send transcript to agent reasoning / LLM pipeline or emit event
        await job_ctx.emit_event("user_input_transcribed", transcript)

async def on_participant_connected(participant: RemoteParticipant, job_ctx: JobContext):
    """
    Subscribe to participant's audio track(s) when they join.
    """
    # Subscribe to each published audio track for the participant
    for publication in participant.tracks:
        if publication.kind != "audio" or not publication.is_subscribed:
            continue

        track = await publication.subscribe()  # obtain RemoteAudioTrack
        # Launch a task to process that participant's audio track independently
        job_ctx.loop.create_task(handle_track(track, participant, job_ctx))

async def job_entrypoint(job_ctx: JobContext):
    """
    Main job entrypoint — enumerate existing participants and hook events for future ones.
    """
    # Subscribe to already-present participants
    for participant in job_ctx.room.participants:
        await on_participant_connected(participant, job_ctx)

    # Listen for future participant connected events
    @job_ctx.on("participant_connected")
    async def _on_connected(new_participant: RemoteParticipant):
        await on_participant_connected(new_participant, job_ctx)

    # Optional: handle participant disconnected to cleanup tasks
    @job_ctx.on("participant_disconnected")
    async def _on_disconnected(left_participant: RemoteParticipant):
        job_ctx.cancel_tasks_for_participant(left_participant.identity)
Key points to adapt in your real implementation • Replace 
is_speech_frame
, 
collect_until_end_of_utterance
, and 
my_stt_client.transcribe
with your chosen VAD/segmenter and STT implementations. • Ensure each per-track task is isolated so you can keep transcripts tied to the correct 
participant.identity
. • Use timestamps from frames or your STT response to keep ordering when merging multi-speaker conversation history. • Emit or store transcripts with identity and timestamp so the LLM prompt can reconstruct speaker turns (e.g., "[Alice] Hello", "[Bob] I agree"). If you’d like, I can: • Provide a more concrete example integrating a specific STT library (please tell me which STT you use), or • Show an example of how to build an LLM prompt that includes speaker tags and timestamps for better context. Which would you prefer?
Open in Slack
PreviousNext
Powered by