How can I run the livekit test & eval setup (with LLM judge) from another api? Like outside of livekit. I wanna be able to programmatically trigger tests.

Thanks for reaching out, Lukas. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

nodejs. Please show an example of how this would work. Imagine just any external server api executing the test. Its like an endpoint we wanna be able to trigger to run the test suite.

Thanks — I’ll show a minimal Node.js example that exposes an HTTP endpoint which programmatically runs a Vitest test file that uses the AgentSession + LLM judge. This example assumes your test file already exists and that required env vars (LLM keys, any LiveKit keys) are set for the process. Files in this example: • server.js — Express endpoint that triggers Vitest programmatically • tests/agent.test.js — a simple Vitest test that runs an AgentSession and calls

judge()

(pseudo-code placeholders where needed) 1) server.js

// server.js
const express = require('express');
const { run } = require('vitest/node');

const app = express();
app.use(express.json());

app.post('/run-tests', async (req, res) => {
  try {
    // Optionally accept env overrides in the request body
    if (req.body.env) {
      Object.assign(process.env, req.body.env);
    }

    // Run Vitest programmatically; specify the test file(s)
    const result = await run({
      // files can be globs or specific paths
      include: ['tests/agent.test.js'],
      reporters: ['default'],
      silent: false,
      // pass other options if needed
    });

    // run() returns a result object; map to a simpler response
    res.json({
      success: result.success,
      details: {
        passed: result.passed,
        failed: result.failed,
        stats: result.stats,
      },
    });
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.listen(3000, () => {
  console.log('Test trigger server listening on port 3000');
});

2) tests/agent.test.js (example using Vitest + LiveKit test utilities)

// tests/agent.test.js
const { test, expect } = require('vitest');
// import your AgentSession and any test helpers
const { AgentSession } = require('your-livekit-test-package-or-local-path');
// import an LLM client (e.g., OpenAI) if your judge uses it
const OpenAI = require('openai');

test('agent responds and judge accepts', async () => {
  // Ensure your provider key is available via env or set here
  const llmApiKey = process.env.OPENAI_API_KEY;
  if (!llmApiKey) throw new Error('OPENAI_API_KEY not set');

  const openai = new OpenAI({ apiKey: llmApiKey });

  // Create an LLM wrapper compatible with your judge() usage
  const llm = {
    // implement a minimal `call` or `generate` function expected by judge()
    async generate(prompt) {
      const resp = await openai.chat.completions.create({
        model: 'gpt-4o-mini', // example
        messages: [{ role: 'user', content: prompt }],
      });
      return resp.choices[0].message.content;
    },
  };

  // Run AgentSession in-process (doesn't need LiveKit server)
  await (async () => {
    const session = new AgentSession({ llm }); // adapt constructor per your package
    await session.start(/* assistant or agent config */);

    const result = await session.run({ user_input: 'Hello' });

    // Example: find assistant message and call judge()
    const assistantMessage = result.events.find(e => e.role === 'assistant' && e.type === 'message');
    expect(assistantMessage).toBeDefined();

    // Call judge (adjust API to match your test helpers)
    const judgeResult = await result.expect.next_event().is_message({ role: 'assistant' }).judge(
      llm,
      { intent: 'Offers a friendly introduction and offer of assistance.' }
    );

    expect(judgeResult.passed).toBe(true);
  })();
});

Notes and tips: • Ensure environment variables (e.g.,

OPENAI_API_KEY

,

LIVEKIT_API_KEY

,

LIVEKIT_API_SECRET

) are set in the process that runs server.js. The endpoint can accept env overrides, but be cautious about security. • The exact imports and method names for

AgentSession

,

judge()

, and LLM wiring may differ depending on your LiveKit test package path; replace

require('your-livekit-test-package-or-local-path')

with your actual module. • The Vitest

run()

call returns an object containing

success

,

passed

,

failed

, and

stats

you can return to callers. • For heavy runs, run the test execution in a worker or background job to avoid blocking the HTTP request; respond immediately with a job id if needed. Would you like a full runnable example wired to a real LiveKit test package path and a complete OpenAI client snippet for the judge()? , with interactive elements

Thanks for the detailed example! One clarification before I go down that path: In your example, the Express server imports

AgentSession

directly from

@livekit/agents

. That means the Express server itself needs to have the

@livekit/agents

SDK installed and must run in a Node.js process that has access to it. So to confirm: there's no way to trigger the test suite remotely against a running LiveKit agent deployment, right? The test runner must live in the same codebase/process that has

@livekit/agents

as a dependency — it's purely in-process, not an API call to LiveKit Cloud or the agent server? Basically: can I run these tests from a completely separate service (e.g. a Convex action or any external API) that does NOT have

@livekit/agents

installed, or do I need to host the test runner where the SDK lives?

Thanks — good question. You're correct. The test & eval runner (including

AgentSession

and the LLM-based

judge

) runs in-process and requires the

@livekit/agents

test library to be available where the runner executes. There is no remote LiveKit API that will run those tests for you against a deployed agent; the tests execute locally (or in whatever process/container) that has the SDK and LLM provider keys. What you can do instead: • Host the test runner as a separate service or container that includes

@livekit/agents

and expose an HTTP endpoint (like the Vitest example). Your external system (Convex action, webhook, CI) can call that endpoint to trigger tests. • Run tests in CI/CD pipelines (GitHub Actions, etc.) that have the SDK installed and return results via webhooks or artifacts. • For non-blocking flows, trigger the runner to enqueue a job and return a job ID, then poll or receive a webhook with results. If you want, I can: • Show a minimal Dockerfile and deployment pattern for a standalone test-runner service. • Give an example of a secure endpoint that accepts requests to run tests and returns results or job IDs. Which would help you more? , with interactive elements

But you said something around env variables. That sounded like we can just define our entire testrunner inside an external api (not in the livekit deployment) - liek convex - access the exact hydrated agent and run our tests. Is that the case or no?

Short answer: no — env vars alone do not let you run the LiveKit test & eval suite against a remote agent deployment. Clarification: • The test runner (code that uses

AgentSession

,

judge()

, etc.) must run where the

@livekit/agents

SDK and its dependencies are installed. It executes in-process and requires those modules to be present. • Environment variables only provide credentials/config (LLM keys, LiveKit keys). They do not provide a remote API that will execute the test logic on a deployed agent for you. • You can host the test runner as a separate service (container, VM, or CI job) that exposes an HTTP endpoint which your external systems (Convex, webhook, CI) call. That external caller does not need

@livekit/agents

— only the test-runner service needs it. If you’d like, I can: • Provide a minimal Dockerfile + compose/example to run the test-runner as a standalone service, or • Show a secure endpoint pattern that accepts test requests and returns job IDs/results. Which of those would help you most? , with interactive elements

but i can run the testrunner speartely from the lk cloud agent deployment? I mean that is all I need. 1. HERE: the lk cloud deployment 2. THERE: The api endpoint I can call from any other server or client that will execute tests against our agent config

Yes — you can host the test runner separately from your LiveKit Cloud agent deployment. Summary/confirmation: • The test & eval framework runs in-process and requires the

@livekit/agents

SDK; the runner must live where that SDK is installed. • You cannot ask LiveKit Cloud to run those tests remotely for you—env vars only supply keys/config, they don't provide remote execution. • So your desired setup is valid: (1) LiveKit Cloud runs your agent, and (2) a separate test-runner service (container/VM/CI job) that has

@livekit/agents

installed exposes an HTTP API which any external system (Convex, webhook, client) can call to execute tests against your agent config. Recommended patterns (brief): • Standalone service: package the test runner +

@livekit/agents

into a container, expose a /run-tests endpoint that runs Vitest/Vitest programmatically and returns results or a job ID. • CI runner: run tests in CI (GitHub Actions, etc.) and trigger via webhook or API; CI returns artifacts/webhook with results. • Job queue: have the endpoint enqueue test jobs and a worker (with the SDK) execute tests to avoid blocking requests. If you’d like, I can provide: • A minimal Dockerfile + docker-compose for a standalone test-runner service, or • A secure Express endpoint example showing job enqueue + result polling/webhook. Which would you prefer? , with interactive elements

how does that separate app access the exact agent/session/main.ts configuration that we have i nthe seprate livekit deployment? like how would that work?

You're asking how a separate test-runner service can run tests against the exact agent/session/main.ts configuration used in your LiveKit Cloud deployment. Short answer: the runner must have access to the agent code/config — it does not magically run that remote code. Here are practical ways to make the runner execute the exact same agent configuration. Key point • Tests run in-process and instantiate the agent code (e.g., AgentSession, handlers, config). The test-runner must import or include the same agent code (main.ts or the module that wires up your agent) in its runtime. Recommended approaches to share the exact agent config • Monorepo ◦ Put agent code and test-runner in the same repository (packages/agent and packages/test-runner). ◦ Test-runner imports the agent package directly (e.g.,

const { createAgent } = require('@yourorg/agent')

) so you run the same codebase. • Publish private package ◦ Package your agent logic as a private npm package (or Git package). ◦ Test-runner installs that package and imports the same exported configuration/constructors. • Git submodule / subtree or repo cloning ◦ Have the test-runner repo include the agent repo as a submodule/subtree or clone the agent code at build time so the runner can import the same files. • Container image of the agent code ◦ Build a container image that contains the agent code (the same main.ts and dependencies). Either: ▪︎ Use that image for the test-runner (adds test tools + Vitest), or ▪︎ Build a separate runner image that copies the agent code into it at build time. How it works in practice (minimal flow) 1. Export agent wiring as an importable module (recommended). ◦ Make sure your agent bootstrap (main.ts) exposes the session/agent factory and config programmatically (e.g.,

createAgentSession(config)

). 1. Test file imports that factory and instantiates an AgentSession in-process: ◦ The test-runner calls

createAgentSession({ /* test overrides */ })

, runs interactions, and calls

judge()

. 1. Test-runner runs in its own process (container/VM/CI) that has

@livekit/agents

and your agent module installed. Notes and tips • Avoid requiring the full server listen/start behavior in tests. Export small, testable factory functions so tests can instantiate the agent session without starting network listeners. • Keep environment-specific differences configurable (pass env overrides or config objects into the factory). • For parity, ensure the same dependency versions are used (lockfiles), or run the runner in the same CI pipeline / build environment as the agent. • Security: do not expose sensitive keys via public endpoints; keep test-runner protected or accept ephemeral credentials only. If you want, I can: • Show a small example of how to export a

createAgentSession

factory from your agent code and import it in a Vitest test, or • Provide a Dockerfile + build steps that copy your agent code into a test-runner image. Which example would you like? , with interactive elements