Thanks — I understand you want to automate tests for voice agents built with Speech-to-Speech (Nova Sonic 2 Realtime). Below is a practical, actionable approach you can use to automate behavior and integration tests while keeping tests fast and cost-efficient. Quick summary of the recommended approach • Use text-mode testing of the realtime model (configure the realtime model with

modalities=["text"]

). This lets you exercise agent logic, intent, tool calls, and context handling without streaming audio in tests. • Run tests inside an async context manager so the

RealtimeModel

and

AgentSession

initialize and clean up correctly. • Validate outputs by asserting on assistant messages, function calls, conversation context, and mocked tool responses. Why text-mode testing works • Speech-to-speech behavior is driven by the same conversation and decision logic as text agents. Testing in text mode verifies intent, conversation state, function/tool calls, and error handling without the overhead of audio encoding/decoding and cost of audio requests. • For end-to-end audio verification you can add a small set of dedicated audio integration tests (see final section). Example pattern (Python, async) 1. Create a

RealtimeModel

with text modality and inject it into an

AgentSession

. 2. Start the agent, run user inputs, and assert expected events (messages, function calls, etc.). 3. Use mocks for external tools to keep tests deterministic. Example test snippet:

async with (
    aws.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
    AgentSession(llm=rt_llm, userdata=userdata) as sess,
):
    await sess.start(YourAgent(userdata=userdata))

    # send a user utterance
    result = await sess.run(user_input="Hello!")

    # expect an assistant message and judge intent
    await result.expect.next_event().is_message(role="assistant").judge(
        rt_llm, intent="Provides a friendly greeting"
    )

    # expect a function/tool call
    await result.expect.next_event().is_function_call(
        name="book_appointment",
        arguments={"date": "tomorrow"}
    )

What to assert in tests • Assistant responses: content, tone, and high-level intent. • Function/tool calls: name and arguments (ensures integrations work). • Context maintenance: follow-up Q/A and memory behavior across messages. • Error handling: invalid input, missing slots, unexpected user behaviors. • Deterministic outputs: mock external dependencies (APIs, DB, calendars). Mocking tools and dependencies • Replace real tools with simple deterministic mocks in unit tests so tests remain fast and reliable:

with mock_tools(YourAgent, {
    "check_availability": lambda date: "Available at 2pm"
}):
    result = await sess.run(user_input="Check availability for Monday")

When you should add audio/E2E tests • Add a small number of integration tests that: ◦ Submit actual audio input (representative cases). ◦ Verify synthesized audio output (e.g., transcription of the TTS output or audio fingerprinting). • Keep these tests limited because they take longer and cost more. Useful references • Using the Testing Framework with Realtime Models: https://livekit.com/field-guides/guide/testing-framework-realtime-models • Testing & evaluation guide: https://docs.livekit.io/agents/start/testing/ • Realtime models overview: https://docs.livekit.io/agents/models/realtime/ Next steps I can help with • Provide a complete runnable test file tailored to your agent (share a small sample of your agent code or the functions you want to assert). • Show an example of an audio integration test (how to send audio and verify output). • Help design test cases for specific behaviors (greetings, booking flow, error scenarios). Which of those would you like me to do next, or can you share a short snippet of your agent (or the functions/tools you use) so I can draft a concrete test example? Sources: Using the Testing Framework with Realtime Models | LiveKit | Testing and evaluation | LiveKit Documentation | Realtime models overview | LiveKit Documentation, with interactive elements