Thanks — yes, you can collect LLM metrics while running automated tests in Python. I’ll walk you through the simplest pattern and show a short example you can drop into your tests. What you can collect • Each test-run emits an

LLMMetrics

event when the LLM produces a response. • Metrics include: Time to First Token (TTFT), total duration, prompt/completion/total token counts, and tokens-per-second. How to capture metrics in your test 1. Create a metrics collector in your test setup. 2. Attach a listener to the session’s

metrics_collected

event. 3. Run the session as usual and assert on behavior; inspect the collected metrics after the run. Example (async Python test)

async def test_agent_with_metrics():
    async with (
        openai.responses.LLM(model="gpt-4o-mini") as llm,
        AgentSession(llm=llm) as session,
    ):
        # Create a simple usage collector
        usage_collector = metrics.UsageCollector()

        # Listen for LLM metrics events and collect them
        session.on("metrics_collected", lambda ev: usage_collector.collect(ev.metrics))

        # Run the session and assert expected events/behavior
        result = await session.run(user_input="Hello")
        await result.expect.next_event().is_message(role="assistant")

        # Access collected metrics
        summary = usage_collector.get_summary()
        print(f"LLM Token Usage: {summary}")

Notes and tips • This works with both standard and realtime LLM models when using the testing framework. • Keep metric collection separate from assertion logic so you can fail tests on behavior while still recording performance data. • If you want aggregated test-suite metrics, persist summaries (e.g., JSON file) after each test and aggregate in a CI step. If you’d like, I can: • Show a variant that logs per-event metrics to a file (JSON or CSV). • Provide an example that fails the test when latency or token usage exceeds thresholds. • Help adapt this to your existing test fixture setup (share a sample fixture or test file). Which would be most helpful? Sources: Testing and evaluation | LiveKit Documentation | LLM Metrics | LiveKit Documentation | Agents (Python) | livekit/livekit_composite | DeepWiki | Using the Testing Framework with Realtime Models | LiveKit