https://livekit.io logo
Join Slack
Powered by
I want to build a crew that can do multiple consid...
# ask-ai
g
I want to build a crew that can do multiple consideration. It's sound like the workflow https://docs.livekit.io/agents/build/workflows but there is a different I want to build: • A single Orchestrator: the only agent will speak to user • A Safety agent: run on all (or almost) messages of the user, check if there is any dangerous intent. • Multiple domain-oriented agents: they never speak to client, they just say what need to say to Orchestrator, so it (Orchestrator) will respond to the user For every message in a conversation, always go to Orchestrator, then it will "asks" other corresponding agent for the answer, then use that answer as the hint, do generate the reply to the user (in behalf of Orchestrator, not the domain agent). Even in the case of the domain-agent needs to add multiple follow-up questions, each message still go through Orchestrator to that agent. Is it a good design? Is it possible? If yes, how can I do it?
t
Thanks for reaching out, Phuong. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Absolutely, your proposed design looks great and fits nicely with what LiveKit can do. Here’s a friendly breakdown of how you can bring it to life: • Orchestrator Agent: Think of this agent as the team captain—it manages the whole workflow and is your main point of contact with the user. It delegates tasks to other agents and puts together the responses based on what they provide. • Safety Agent: This one’s like your safety net. It checks user messages to make sure everything’s safe and flags any risky or harmful intent. You can have it review all messages or just the ones you choose. • Domain-Oriented Agents: These agents are your specialists, each handling specific tasks and sending their results back to the Orchestrator, who then crafts the final reply for the user. Here’s how you can get started: 1. Extend the 
Agent
class to create your Orchestrator, Safety, and Domain agents. 2. Kick off the session with the Orchestrator as your starting agent. 3. Use 
function_tool
so the Orchestrator can smoothly switch between agents depending on what the user needs or the context. 4. Keep the conversation flowing by sharing info across agents using 
context.userdata
. 5. Make sure all user messages go through the Orchestrator—it’ll handle delegating tasks and putting together the final responses. This setup helps keep things organized, safe, and smooth for your users. If you’d like, I’m happy to share some code examples or help you out further—just let me know! Sources: Building voice agents | LiveKit Docs | Workflows | LiveKit Docs | examples/voice_agents/multi_agent.py
g
5. Make sure all user messages go through the Orchestrator—it’ll handle delegating tasks and putting together the final responses.
This is the main concern that I'm not sure can do with livekit. The workflow and handoff logic of LiveKit is great, but it seems not do-able if I don't want domain-agents to speak, they just return their payload to Orchestrator so Orchestrator can speak. Do you have any sample implementation to do so?
t
To make sure all user messages go through the Orchestrator and that domain agents only send back payloads for the Orchestrator to handle the response, here’s a simple way to set it up: 1. Set Up the Orchestrator: Think of the Orchestrator as the main coordinator. It’ll take in all user messages and then pass tasks along to the domain agents. 2. Use AgentSession: Start an 
AgentSession
with the Orchestrator as the first point of contact. This way, all communication starts and ends with the Orchestrator. 3. Domain Agents Return Payloads: Have domain agents work on their tasks and send their outputs (payloads) back to the Orchestrator instead of replying directly to the user. The Orchestrator then uses these outputs to craft the final response. 4. RunResult Handling: Use 
RunResult
to gather responses from the domain agents. The Orchestrator can then combine these and create a smooth, unified reply for the user. 5. Workflow Configuration: Set up your workflows so that every handoff between agents loops back to the Orchestrator for the final processing and response. This way, the Orchestrator stays as the main point of contact with your users, while the domain agents quietly do their part behind the scenes. If you’d like, I’m happy to share a detailed code example for any part of this—just let me know! Sources: Agents Core Architecture | livekit/livekit_composite | DeepWiki | livekit-agents/livekit/agents/llm/_provider_format/anthropic.py
g
Yes, please. A detailed code will help a lot.
3. Domain Agents Return Payloads: Have domain agents work on their tasks and send their outputs (payloads) back to the Orchestrator instead of replying directly to the user. The Orchestrator then uses these outputs to craft the final response.
4. RunResult Handling: Use 
RunResult
to gather responses from the domain agents. The Orchestrator can then combine these and create a smooth, unified reply for the user.
5. Workflow Configuration: Set up your workflows so that every handoff between agents loops back to the Orchestrator for the final processing and response.
t
I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!
g
In short: • LiveKit has several approaches, but no native ability to let domain-agent to "pause" the natural response and handoff to Orchestrator (handoff only works in function_tool call) • We can re-design the architecture to achieve the goal (I don't have idea in mind yet) Hope that the LiveKit team can provide the ability to "pause" the agent speak and just return its text-response to handoff agent so the handoff agent can handle it as the main speaker. Using the single speak agent is significant improved the quality of the response since I can customize its persona carefully where other sub-agent must focus on its domain-related job.
🙌 1
🙏 1
Open in Slack
PreviousNext
Powered by