https://livekit.io logo
Join Slack
Powered by
I have a livekit voice agent and would like to do ...
# ask-ai
p
I have a livekit voice agent and would like to do automated testing and validation on agent response against the list of Q&A pairs I have.. What is the best approach or framework available for this ?
✅ 1
t
Thanks for reaching out, mithun. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great question. You want to automatically test a LiveKit voice agent’s responses against a list of Q&A pairs. Below I’ll outline the recommended approaches, when to use each, and practical next steps you can take. Short answer (recommended) • Use LiveKit Agents’ built-in testing framework (text-only testing) to validate responses against your Q&A pairs. It’s cost‑effective, supports pytest/Vitest, and includes LLM-based judgment helpers to compare agent outputs to expected intents. Why the built-in testing framework is usually best • Text-only tests let you simulate conversation turns without running audio, making tests faster and cheaper. • You can programmatically feed each question from your Q&A list into an AgentSession and assert the agent’s outputs. • The framework provides tools to inspect events (e.g., 
next_event().is_message()
), assert function calls/outputs, and run LLM-based judgments (
.judge(llm, intent="...")
) for qualitative matching beyond exact-string equality. • Test helpers work with realtime models in text mode, so you can validate expected behavior even if your production agent uses realtime audio models. When to consider agent-to-agent testing • If you want stronger, automated end-to-end conversational evaluation without human input, create an evaluation agent that plays the user role. • Configure room input options to allow agent input and have the evaluation agent send your Q&A items and judge responses automatically. When to use third-party tools (audio-level testing) • If you need to validate the full audio pipeline (ASR quality, TTS output, latency, audio artifacts) use an end-to-end voice testing provider: ◦ Bluejay, Cekura, Coval, or Hamming — these services simulate real audio calls at scale and provide monitoring/metrics across many permutations. • Use them only if audio-level fidelity and production monitoring are required; otherwise text testing covers conversational logic and intent validation. Practical test plan to get started 1. Prepare your Q&A list in a structured format (CSV/JSON) with fields: id, user_question, expected_intent_or_response, tolerance (exact vs. semantic). 2. Implement a text-only test harness using the Agents testing helpers: a. Create an AgentSession for the agent under test. b. For each Q&A: ▪︎ Call 
await session.run(user_input="...")
▪︎ Inspect the next event: 
result.expect.next_event().is_message(role="assistant")
▪︎ Use 
.judge(llm, intent="...")
for semantic checks or assert on message text for exact-match cases. 1. Run tests in CI (GitHub Actions) using secrets for any LLM API keys the judge needs. 2. Add failure metrics and reporting: pass/fail, confidence from the LLM judge, and example mismatches for debugging. 3. If you want to simulate more realistic dialogues, group Q&A items into multi-turn scenarios and validate stateful behavior. If you want an example • I can show a small pytest example that reads a JSON list of Q&A and runs the session + 
.judge(...)
assertions. • Or I can show how to build an evaluation agent that runs agent-to-agent tests inside a LiveKit Room. Which would you like? • A short pytest example for text-only validation (recommended first step), OR • A sample agent-to-agent test setup to simulate interactive evaluation, OR • Guidance on choosing/configuring a third-party audio testing provider for end-to-end audio validation Pick one and I’ll provide the step-by-step example and code snippets. Sources: Testing and evaluation | LiveKit Documentation | Agents (Python) | livekit/livekit_composite | DeepWiki | examples/voice_agents | livekit/livekit_composite | DeepWiki | examples/frontdesk | livekit/livekit_composite | DeepWiki | LiveKit's Series C: Towards the voice-driven era of computing | LiveKit, with interactive elements
p
can you give a simple script that takes list of q&a validation pairs and ask the same to the livekit agent. Assume livekit agent is accessible in local
t
Thanks — I’ll give you a simple, runnable Python script that reads a list of Q&A pairs and asks each question to a locally running LiveKit agent using the Agents testing helpers. This script uses text-only interaction (no audio), which is fast and recommended for validating conversational behavior. What this does • Loads Q&A pairs from a JSON file (or inline list). • Starts an AgentSession and sends each question with 
session.run(user_input=...)
. • Uses an LLM-based judge to compare the agent response to an expected intent/response description. • Prints pass/fail with feedback for each case. Requirements • Python 3.8+ • Install the required package(s) (adjust package name if your environment differs): pip install "livekit-agents[silero]" python-dotenv • Your agent implementation accessible locally (example assumes an 
Assistant
class you provide). Files 1) test_cases.json (example)
Copy code
[
  {
    "id": "greeting-1",
    "question": "Hello, how are you?",
    "expected_intent": "Responds warmly and offers assistance."
  },
  {
    "id": "weather-1",
    "question": "What's the weather in Tokyo?",
    "expected_intent": "Explains it cannot check live weather or asks for location details."
  },
  {
    "id": "joke-1",
    "question": "Tell me a joke.",
    "expected_intent": "Tells a short, friendly joke."
  }
]
2) validate_agent.py
Copy code
import asyncio
import json
from pathlib import Path

# Imports from LiveKit testing helpers
from livekit.agents import AgentSession
from livekit.plugins import inference

# Replace with your actual agent class/module
# Example: from my_agent import Assistant
from your_agent_module import Assistant

TEST_FILE = Path("test_cases.json")

async def run_validation():
    # Use an LLM for judgment (can be any supported LLM)
    llm = inference.LLM(model="openai/gpt-4o-mini")  # adjust model/provider as needed

    # Load test cases
    cases = json.loads(TEST_FILE.read_text())

    # Start a session and the local agent
    async with AgentSession(llm=llm) as session:
        await session.start(Assistant())  # assumes Assistant is your Agent subclass

        all_passed = True

        for case in cases:
            qid = case.get("id", "<no-id>")
            question = case["question"]
            expected = case.get("expected_intent", "")

            print(f"\n🔍 Test {qid}: '{question}'")
            # Send text-only user input
            result = await session.run(user_input=question)

            # Inspect next event for an assistant message and ask the LLM judge to evaluate it
            judged = await result.expect.next_event().is_message(role="assistant").judge(
                llm, intent=expected
            )

            if judged.passed:
                print(f"✅ Passed: {judged.feedback}")
            else:
                all_passed = False
                reason = getattr(judged, "reason", "No reason provided")
                print(f"❌ Failed: {reason}")
                print(f"  Judge feedback: {getattr(judged, 'feedback', '')}")

        if all_passed:
            print("\n🎉 All tests passed.")
        else:
            print("\n⚠️ Some tests failed. See output above for details.")

if __name__ == "__main__":
    asyncio.run(run_validation())
How to adapt this to your environment • Replace 
from your_agent_module import Assistant
with your agent class. The 
Assistant
should be the agent you run locally (subclass of Agent). • Change the judge LLM model string if you use another provider or model. • For strict matching instead of LLM judgment, replace the 
.judge(...)
call with a direct text check:
Copy code
msg = await result.expect.next_event().is_message(role="assistant").get_text()
if expected_exact == msg.strip():
    passed = True
• To run multi-turn scenarios, send subsequent 
session.run(user_input=...)
calls in sequence and assert on the sequence of events. CI tips • Use secrets for any LLM API keys. • Run this script in CI to validate regressions on each push. • Collect judge feedback and failure examples for debugging. If you want, I can: • Provide a pytest-style test wrapper for CI integration. • Convert this to a Node.js example. • Show a CSV-driven loader instead of JSON. Which would you like next? Sources: Testing and evaluation | LiveKit Documentation, with interactive elements
p
is this the right approach to validate a faq agent ? where input Q&A are fixed
t
Yes — that is the right approach for validating a FAQ-style agent whose Q&A pairs are fixed. Your text-only testing harness is appropriate, efficient, and aligned with best practices. Here’s why and a few practical tips to make it robust. Why this is a good fit • Fast and inexpensive: Text-only tests avoid audio encoding/decoding and are much cheaper and faster to run than end-to-end audio tests. • Deterministic checks: For FAQ-style answers you can often use exact-match or normalized-string checks (or regex) when responses are expected to be stable. • Semantic flexibility: Where answers may vary phrasing, an LLM-based judge lets you evaluate whether the response fulfills the expected intent or content rather than requiring exact wording. • CI friendly: Runs well in automated pipelines to catch regressions when you change agent logic, knowledge sources, or model versions. Best practices and suggestions • Use the right comparison strategy per question: ◦ Exact match or canonical answer for strict FAQs (e.g., policy text, phone number). ◦ Normalized text comparisons (strip whitespace, lowercasing, remove punctuation) for minor variations. ◦ Regex or key-phrase checks when you expect a set of acceptable phrasings. ◦ LLM-based judgment for semantic equivalence when phrasing can vary widely. • Include metadata per test case: ◦ id, question, expected_intent (for judge), expected_exact (for strict checks), tolerance level (exact/semantic), and any required context/state. • Add negative tests and edge cases: ◦ Questions outside FAQ scope to verify graceful fallback (e.g., “I don’t know” behavior). ◦ Ambiguous or malicious prompts to check safety/guardrails. • Test multi-turn context where needed: ◦ If some FAQs require follow-up questions (clarification), include multi-turn scenarios to validate stateful behavior. • Monitor judge confidence and failures: ◦ Log judge reasons/feedback and sample failing outputs for quick debugging. ◦ If using an LLM judge, record its confidence or explanation to prioritize failures. • Keep canonical answers up to date: ◦ Maintain a single source of truth (JSON/CSV) for canonical FAQ answers so tests and production share the same data. When to add audio-level testing • Add third-party audio E2E tools (Bluejay/Cekura/Coval/Hamming) only if you must validate ASR/TTS accuracy, audio latency, or production audio artifacts. • Otherwise, start with text-only and add audio tests selectively for high-risk flows. Short checklist to get the most value • Classify each Q&A as exact vs. semantic. • Choose exact-match, regex, or LLM-judge per case. • Add negative/edge-case tests. • Run tests in CI and store failing transcripts. • Periodically re-run tests after knowledge base updates. If you want, I can: • Convert your Q&A list into a sample JSON with metadata fields (exact vs semantic). • Provide a pytest wrapper that implements both exact and LLM-based checks and outputs a CSV of failures for tracking. Which of those would be most helpful next? , with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by