Hi @future-ocean-83972! You can checkout https://www.cekura.ai/ I am the co-founder of Cekura. It has the functionalities to simulate any kind of conversation and also evaluate it on transcript + audio metrics. It has comes with a free trial. I am the co-founder of Cekura, lmk if any doubts

Thanks — appreciate you sharing Cekura and the link, and congrats on the product. I’ll keep the focus on a concrete, runnable Python example for LiveKit (two agents: AgentUnderEvaluation and AgentSimulatingAUser), a 3–5 turn scripted dialogue where the simulating agent reports a problem, and a simple automated evaluation that asserts whether the problem was resolved. Below is a concise Python-style example you can adapt to your environment. It uses the LiveKit agent/session/testing helpers shown previously — I’ve replaced the illustrative helper names with clear placeholders and shown where to plug in your real SDK calls or helper functions. • Replace token creation, client initialization, and session/message APIs with the exact functions from your LiveKit SDK. • The evaluation here is a simple predicate that looks for an explicit resolution confirmation in the transcript; you can swap that for an LLM-based judge or audio/transcription metric checks as needed.

# Example: create a LiveKit room with two agents, run a 5-turn dialogue,
# and evaluate whether the user's problem was resolved.
# Replace placeholders with your LiveKit SDK calls and credentials.

import asyncio
from livekit import rtc  # placeholder import; use actual SDK module names
# Placeholder helper classes — replace with your actual session/test helpers
from livekit.agents import RoomOptions, RoomConfiguration, RoomAgentDispatch
# Assume you have an AgentSession-like helper to start an agent and emit messages
# If not, replace session.emit_* with your SDK-specific publish/send APIs.

LIVEKIT_API_KEY = "YOUR_API_KEY"
LIVEKIT_API_SECRET = "YOUR_API_SECRET"
ROOM_NAME = "eval_room_001"
AGENT_UNDER_EVAL = "AgentUnderEvaluation"
AGENT_SIM_USER = "AgentSimulatingAUser"

def create_token_with_room_config():
    """
    Build an access token that includes a room configuration to dispatch both agents.
    Replace this logic with your actual AccessToken building code.
    """
    token = AccessToken(LIVEKIT_API_KEY, LIVEKIT_API_SECRET)  # placeholder
    room_cfg = RoomConfiguration(
        agents=[
            RoomAgentDispatch(agent_name=AGENT_UNDER_EVAL),
            RoomAgentDispatch(agent_name=AGENT_SIM_USER),
        ]
    )
    token.with_room_config(room_cfg)
    token.with_room(ROOM_NAME)
    return token.to_jwt()

# Scripted 5-turn dialogue
SCRIPT = [
    {"speaker": AGENT_SIM_USER, "text": "Hi, I'm having trouble connecting my headset to the app — there's no audio."},
    {"speaker": AGENT_UNDER_EVAL, "text": "Sorry about that. Which headset model and OS are you using?"},
    {"speaker": AGENT_SIM_USER, "text": "It's the Acme Headset X on macOS 13."},
    {"speaker": AGENT_UNDER_EVAL, "text": "Thanks. Please open Settings > Audio and select 'Acme Headset' for both input and output. Then restart the app."},
    {"speaker": AGENT_SIM_USER, "text": "I selected it and restarted — I can hear audio now. Problem solved, thank you!"}
]

def simple_resolution_predicate(transcript_events):
    """
    Return True if the simulated user explicitly confirms resolution.
    Looks for resolution phrases + topic keywords.
    """
    full = " ".join(ev["text"].lower() for ev in transcript_events)
    solved_phrases = ["problem solved", "it works now", "i can hear", "fixed", "resolved", "thank you"]
    topic_keywords = ["headset", "audio", "sound", "connect"]
    return any(p in full for p in solved_phrases) and any(k in full for k in topic_keywords)

async def run_simulation():
    token = create_token_with_room_config()

    # Start a session that allows agent-to-agent communication (ensure agent participant kinds allowed)
    session = AgentSession(  # placeholder helper — replace with actual session creation
        token=token,
        room=ROOM_NAME,
        room_options=RoomOptions(participant_kinds=[rtc.ParticipantKind.PARTICIPANT_KIND_AGENT])
    )

    # Start the agent under evaluation (this should dispatch both agents into the room)
    await session.start(agent_name=AGENT_UNDER_EVAL)

    transcript = []

    for turn in SCRIPT:
        speaker = turn["speaker"]
        text = turn["text"]

        if speaker == AGENT_SIM_USER:
            # Simulating user input from AgentSimulatingAUser
            # Replace with the SDK call that sends a user_input or participant message as that agent
            await session.emit_user_input(text, from_agent=AGENT_SIM_USER)  # placeholder
        else:
            # Either let the real agent produce the response, or emit the scripted agent response
            # If the agent generates the reply, you may await a response event instead of emitting.
            await session.emit_agent_message(text, agent_name=AGENT_UNDER_EVAL)  # placeholder

        transcript.append({"speaker": speaker, "text": text})
        await asyncio.sleep(0.2)

    # Clean up
    await session.stop()

    # Automated evaluation
    solved = simple_resolution_predicate(transcript)

    print("Transcript:")
    for ev in transcript:
        print(f"{ev['speaker']}: {ev['text']}")

    print("\nEvaluation result:")
    if solved:
        print("Outcome: The user's problem appears to be SOLVED.")
    else:
        print("Outcome: The user's problem does NOT appear to be solved.")

if __name__ == "__main__":
    asyncio.run(run_simulation())

Next steps I can take for you • Convert placeholders to the exact LiveKit SDK calls you use (token builder, session start, message publish). If you share which LiveKit client/testing library and versions you’re using (or paste a short snippet showing how you start an agent session and send messages), I’ll produce a fully concrete script. • Swap the simple predicate for an LLM-based judge or add audio/transcription metrics (e.g., verify transcription confidence or presence of a confirmation phrase in the ASR output). I can show an example using an LLM judge if you want. • Add assertions and test harness behavior (pass/fail exit codes) for CI integration. Would you like me to replace the placeholders with code matching your SDK and environment? If so, tell me which LiveKit SDK / testing helpers you use or paste the minimal code you currently use to start agents and send messages.