Can we have single agent with multiple users in th...
# ask-ai
l
Can we have single agent with multiple users in the room and can that agent listen to all the participants and respond or call tools, like Notetaker?
t
Thanks for reaching out, ROHIT. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Absolutely! A single LiveKit agent can comfortably interact with multiple users in a room, listen to everyone at once, and even use handy tools like a Notetaker. This is all made possible by the LiveKit Agents framework. The agent joins the room as a special participant and tunes into the audio tracks from all users. It can then process what it hears using tools like Speech-to-Text (STT) engines and respond through Text-to-Speech (TTS) or other ways. Tools like a Notetaker can be added as function tools or plugins, helping the agent summarize conversations or handle other helpful tasks. For instance, the KITT agent listens to multiple users, summarizes the discussion, and takes notes seamlessly. If you're curious, you can check out the implementation details and examples in the KITT GitHub repository or the LiveKit Agents Documentation. Would you like me to share a complete working code template for a multi-user Notetaker agent? I’m here to help! Sources: Rooms, participants, and tracks | LiveKit Docs | Building voice agents | LiveKit Docs | Live Conversations with AI using ChatGPT and WebRTC | README.md | Remote method calls (RPC) | LiveKit Docs
l
yes, can you give me the code for python sdk for agent worker and react components for frontend if required
t
I'm happy to share the Python SDK code for an agent worker along with the React components for the frontend. Let's walk through it together: Python Agent Worker Code
Copy code
from livekit import agents, JobContext, JobRequest, WorkerOptions
from livekit.plugins import openai

async def entrypoint(ctx: JobContext):
    # Connect to the room with audio subscription only
    await ctx.connect(auto_subscribe=agents.AutoSubscribe.AUDIO_ONLY)

    # Create a multimodal agent using OpenAI's Realtime model
    agent = agents.MultimodalAgent(
        model=openai.models.RealtimeModel(
            instructions="You are a helpful assistant.",
            voice="alloy",
            temperature=0.8,
            modalities=["text", "audio"],
        )
    )
    # Start the agent in the room
    agent.start(ctx.room)

if __name__ == "__main__":
    # Run the worker with the entrypoint function
    agents.cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
React Frontend Components Session View Component
Copy code
'use client';

import React, { useEffect, useState } from 'react';
import { AnimatePresence, motion } from 'motion/react';
import {
  type AgentState,
  type ReceivedChatMessage,
  useRoomContext,
  useVoiceAssistant,
} from '@livekit/components-react';
import { AgentControlBar } from '@/components/livekit/agent-control-bar/agent-control-bar';
import { ChatEntry } from '@/components/livekit/chat/chat-entry';
import { ChatMessageView } from '@/components/livekit/chat/chat-message-view';
import { MediaTiles } from '@/components/livekit/media-tiles';
import useChatAndTranscription from '@/hooks/useChatAndTranscription';

function isAgentAvailable(agentState: AgentState) {
  return agentState == 'listening' || agentState == 'thinking' || agentState == 'speaking';
}

export const SessionView = ({ disabled, capabilities, sessionStarted, ref }) => {
  const { state: agentState } = useVoiceAssistant();
  const [chatOpen, setChatOpen] = useState(false);
  const { messages, send } = useChatAndTranscription();
  const room = useRoomContext();

  async function handleSendMessage(message: string) {
    await send(message);
  }

  useEffect(() => {
    if (sessionStarted) {
      const timeout = setTimeout(() => {
        if (!isAgentAvailable(agentState)) {
          room.disconnect();
        }
      }, 10_000);
      return () => clearTimeout(timeout);
    }
  }, [agentState, sessionStarted, room]);

  return (
    
  );
}
To get you started smoothly: • Create the agent by running
lk app create --template agent-starter-python
. • Set up the frontend with
lk app create --template agent-starter-react
. • Once that's done, just install the dependencies and run both applications. If you have any questions or need a hand with anything, please don’t hesitate to ask—I’m here to help! Sources: OpenAI and LiveKit partner to turn Advanced Voice into an API | complex-agents/role-playing/role_playing_frontend/components/livekit/agent-... | AI Voice Assistant Quickstart | LiveKit Docs | OpenAI and LiveKit partner to turn Advanced Voice into an API