https://livekit.io logo
Join Slack
Powered by
We are building a chat agent using LiveKit agent. ...
# ask-ai
b
We are building a chat agent using LiveKit agent. How can we implement A/B test of an answer by agent? One agent using tool A v.s. the other using tool B.
✅ 1
t
Thanks for reaching out, Masayuki. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — you're asking how to A/B test two agent answer strategies (one using Tool A, the other Tool B). I can help — here’s a concise plan you can implement. 1) Create two agent variants • Deploy two agents with distinct names, e.g. 
chat-agent-tool-a
and 
chat-agent-tool-b
. • Configure each agent to include only its intended tool set (
tool_a
for the first, 
tool_b
for the second). 2) Route traffic for the A/B split • When issuing the room access token or creating the room session, set the room’s agent dispatch to the chosen agent name. Randomize assignment server-side to produce your desired split (50/50, 70/30, etc.). Example: include 
RoomAgentDispatch(agent_name="chat-agent-tool-a")
or 
"chat-agent-tool-b"
in the room configuration. 3) Keep isolation and reproducibility • Use separate agent names and separate deployments so each variant runs independently (this avoids state/tool bleed between variants). 4) Collect metrics and evaluate • Capture transcripts, response times, tool usage, and any custom signals (user ratings, clicks, conversion events). • Compare success metrics across variants (accuracy, latency, user satisfaction). Optionally run an evaluation agent to score responses automatically. If you want, I can: • Provide a short code snippet for creating tokens with the room dispatch setting in your language. • Suggest which metrics to capture for statistical significance. Which of those would you like next, and what language or runtime are you using? Sources: Avoiding crosstalk when multiple developers share a LiveKit project | LiveK... | Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit | Testing and evaluation | LiveKit Documentation, with interactive elements
b
I only want the A/B test in one turn of the chat conversation. Just like when agent needs to use the tool A, we want the agent to generate another response in parallel using tool B.
t
Thanks — you're asking how to run an A/B test inside a single chat turn where the agent calls Tool A and Tool B in parallel and returns both responses for comparison. I can help — quick plan and a couple of questions so I can give a precise example. What you can do • Enable parallel tool calls for the LLM plugin you use (the 
parallel_tool_calls
parameter). This lets the model invoke multiple tools in the same turn. • Register both tools with the agent and ensure they return comparable outputs (same format/schema). • Trigger both tools in the same agent turn so the agent receives both results and/or you capture both raw tool outputs. • Store the two responses (tool A result and tool B result) with metadata (timestamps, tool name, request id) for later evaluation. • Compare automatically (e.g., scoring function, evaluation agent) or show both to users and collect preference/ratings. Limitations to note • 
parallel_tool_calls
must be supported by your chosen LLM provider; not all models/plugins support it. • There’s no built-in LiveKit mechanism that automatically does A/B scoring—you must capture and evaluate outputs yourself or via an evaluation agent. Questions so I can give code/config tailored to you 1. Which LLM plugin/provider are you using (Telnyx, Groq, Together, etc.)? 2. Which runtime/language do you want the example in (Python, Node)? 3. Do you want the agent to (a) return both responses to the user, (b) return one selected response after internal scoring, or (c) return one and log the other for offline analysis? Pick the provider and one of options (a/b/c) and I’ll give a focused code/config snippet and an example of how to capture/evaluate the two outputs. Sources: Telnyx LLM plugin guide | LiveKit Documentation | Groq LLM plugin guide | LiveKit Documentation | OVHCloud LLM plugin guide | LiveKit Documentation | Perplexity LLM plugin guide | LiveKit Documentation | Together AI LLM plugin guide | LiveKit Documentation, with interactive elements
3 Views
Open in Slack
PreviousNext
Powered by