I'm using Unity. I'm creating a LiveKit room usin...
# ask-ai
b
I'm using Unity. I'm creating a LiveKit room using tokens provided by my own server. Now I want an agent to join my existing room, using another token from my server. What Unity code do I need for this?
t
Thanks for reaching out, Mat. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I understand you’re using Unity as a client and want an “agent” to join that same LiveKit room using a token from your server. I’m happy to clarify how this works and what code you actually need. Short answer • Unity clients join rooms with the Unity LiveKit SDK using tokens from your server. • Agents are not typically run inside Unity. Agents are server-side processes that join rooms using the LiveKit Agents SDK (Node or Python) or the LiveKit server APIs. You do not need (and usually cannot use) Unity C# to run an agent. What you should do instead • If your goal is a human or bot participant inside the room and it’s running inside your Unity app: generate a token on your server and have your Unity client connect using the Unity SDK. • If your goal is an automated agent (for answering calls, recording, moderation, bots): implement that agent as a backend process using the LiveKit Agents SDK (Python/Node) or a server-side client that connects with a token. The agent runs independently from the Unity app and connects to the same room. Unity client example (what Unity needs) Below is the minimal Unity-side flow you’ll use to join a room with a token your server issues: 1. Request a token from your server for the room and participant identity. 2. Use the Unity LiveKit client SDK to connect to the room with that token. Example pseudocode (Unity C# using Unity LiveKit SDK patterns):
Copy code
// 1. Fetch token from your server (HTTP GET/POST)
string token = await FetchTokenFromServer(roomName, participantName);

// 2. Create a new Room and connect
var room = new Room();
// Optional: configure room options here

// Connect using the token (this call is asynchronous in the SDK)
await room.Connect("wss://<your-livekit-url>", token);

// Now you are joined to the room. Publish tracks or subscribe to remote tracks as needed.
If you want, I can give a concrete snippet matching the exact Unity LiveKit SDK methods you’re using (for example,
Room.Connect
signature, track creation, or handling remote participants). Tell me which Unity LiveKit package/version you have and whether you’re targeting WebGL or native Unity. Agent-side guidance (server) • Implement the agent as a backend service using the Agents SDK (Python or Node) or a server LiveKit client. • Generate a token for the agent on your server (same approach as for Unity), and start the agent process to connect to the room using that token. • The agent will operate independently and can publish/subscribes just like any client. Next steps I can help with • Provide a concrete Unity C# code snippet targeting the exact Unity LiveKit SDK version you’re using. • Show how to implement a simple agent connection using the Python or Node Agents SDK (sample code). • Help with your server token generation endpoint (what fields to include in the token request for room, identity, and permissions). Which would you like next — a full Unity C# example with actual SDK method names, or a sample agent implementation (Python/Node) to run on your server? Also tell me the Unity LiveKit SDK version you’re using. Sources: Server startup modes | LiveKit Documentation | Agent dispatch | LiveKit Documentation | Agents (Python) | livekit/livekit_composite | DeepWiki | Real-time audio and video in the Metaverse | Simple Call Answering Agent | LiveKit Documentation
b
I understand that for my Unity code to trigger the dispatch of an agent to an existing room I should use livekit.AgentDispatchService/CreateDispatch. If I've got a participant token from my server, what C# code do I use to call CreateDispatch?
t
Thanks — I’m responding to your question about calling livekit.AgentDispatchService/CreateDispatch from Unity using a participant token. Short answer • You should not call AgentDispatchService.CreateDispatch from Unity with a participant token. • Agent dispatch is a server-side administrative action that requires privileged credentials (API key/secret). Dispatching an agent from a client-side app is insecure and not supported. Why not • CreateDispatch is part of the agent dispatch server API and requires server credentials (LIVEKIT_API_KEY / LIVEKIT_API_SECRET). Those credentials must be kept secret and never embedded in a Unity client. • A participant token (the token your Unity client uses to join a room) does not have the privileges to call administrative APIs like AgentDispatchService.CreateDispatch. What to do instead (recommended flow) 1. From your Unity client, call a secure endpoint on your server (authenticated as you require) to request an agent dispatch. Send any necessary context (room name/id, participant identity, optional metadata). 2. On your server, call AgentDispatchService.CreateDispatch using your server-side SDK or REST/GRPC API with your API key/secret. The server is the only safe place to perform this action. 3. The server returns a safe result to Unity (e.g., success/failure, agent id, status). Unity continues normal room behavior using participant tokens as usual. If you want, I can provide: • A concrete C# example showing how to implement the server-side CreateDispatch call (for a .NET server), including the request body and using the proper credentials. • A minimal Unity C# snippet showing how to call your own secure server endpoint to request dispatch (no privileged credentials in the client). • Example payloads (what to send from Unity to your server and what the server sends to the LiveKit dispatch API). Which would be most helpful: a server-side C# (.NET) CreateDispatch example, or the Unity client code that calls your server endpoint? Also tell me whether you’re using REST/gRPC on the server or a specific LiveKit server SDK, and I’ll tailor the sample code. Sources: Agents telephony integration | LiveKit Documentation | Agent dispatch | LiveKit Documentation | Real-time audio and video in the Metaverse
b
My server is in JavaScript. I could use a JavaScript function to dispatch an agent, specified by agent_name, to a specified room.
t
Great — since your server is JavaScript, you should perform the agent dispatch from the server using your LiveKit API key/secret. Below is a clear server-side example showing how to call AgentDispatchService.CreateDispatch (via the LiveKit server SDK or via the REST/gRPC endpoint). The Unity client calls your server endpoint (not shown here) to request the dispatch. Notes before code • Do not call CreateDispatch from the Unity client. Keep your API key/secret on the server only. • The server will authenticate with LiveKit using your API key/secret and request an agent be dispatched to the specified room and agent_name. • Adjust fields (room, agent_name, metadata) to fit your use case and LiveKit server URL. Option A — Using the LiveKit Node server SDK (recommended if available) (If you have the official LiveKit Node server SDK installed; adapt imports to your package name/version.)
Copy code
// Example: node server using LiveKit server SDK
// npm install @livekit/server-sdk  (example package name; adjust if different)

const {AgentDispatchServiceClient, CreateDispatchRequest} = require('@livekit/server-sdk');
const grpc = require('@grpc/grpc-js'); // if SDK uses grpc under the hood

// Configure
const LIVEKIT_URL = 'your-livekit-server:443'; // host with port (no scheme)
const API_KEY = process.env.LIVEKIT_API_KEY;
const API_SECRET = process.env.LIVEKIT_API_SECRET;

// Create client (SDK may provide helper; using raw gRPC client pattern shown)
const metadata = new grpc.Metadata();
metadata.set('authorization', `Basic ${Buffer.from(`${API_KEY}:${API_SECRET}`).toString('base64')}`);

// If the SDK exposes a client wrapper, use that. Example generically:
const client = new AgentDispatchServiceClient(LIVEKIT_URL, grpc.credentials.createSsl());

async function createDispatch(agentName, roomName, metadataFields = {}) {
  return new Promise((resolve, reject) => {
    const req = new CreateDispatchRequest();
    req.setRoom(roomName);
    req.setAgentName(agentName);

    // Attach metadata as JSON string or structured fields per API
    if (Object.keys(metadataFields).length) {
      req.setMetadata(JSON.stringify(metadataFields));
    }

    client.createDispatch(req, metadata, (err, resp) => {
      if (err) return reject(err);
      resolve(resp);
    });
  });
}

// Example usage:
createDispatch('my-agent', 'my-room', {initiatedBy: 'unity-client'})
  .then(res => console.log('Dispatch created:', res))
  .catch(err => console.error('Dispatch error:', err));
Option B — Call the LiveKit server REST/gRPC API directly using fetch (REST) or gRPC (if REST is available) (If the server exposes a REST endpoint for agent dispatch; if not, use the Node SDK or gRPC as above.)
Copy code
// Example using REST POST (if LiveKit exposes HTTP endpoint for dispatch).
// Replace with the correct LiveKit dispatch endpoint path for your deployment.

const fetch = require('node-fetch');

async function createDispatchREST(agentName, roomName, metadataFields = {}) {
  const url = `<https://your-livekit-host/api/dispatch`;> // adjust path to real endpoint
  const body = {
    room: roomName,
    agent_name: agentName,
    metadata: metadataFields
  };

  const resp = await fetch(url, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      // Basic auth using API key:secret — keep secret on server
      'Authorization': 'Basic ' + Buffer.from(`${process.env.LIVEKIT_API_KEY}:${process.env.LIVEKIT_API_SECRET}`).toString('base64'),
    },
    body: JSON.stringify(body),
  });

  if (!resp.ok) {
    const text = await resp.text();
    throw new Error(`Dispatch failed: ${resp.status} ${text}`);
  }
  return resp.json();
}

// Example usage:
createDispatchREST('my-agent', 'my-room', {initiatedBy: 'unity-client'})
  .then(res => console.log('Dispatch REST created:', res))
  .catch(err => console.error(err));
What I need from you to tailor the snippet • Are you using the official LiveKit Node server SDK? If so, tell me the package name/version and I’ll provide exact import/usage. • If not using the SDK, do you prefer a gRPC or REST example? If REST, confirm the correct LiveKit dispatch endpoint on your deployment. • Any specific fields you want sent with the dispatch (e.g., participant identity, call metadata, routing hints)? Next steps I can help with • Provide the exact Node.js code using your LiveKit SDK/package version. • Show how to expose a secure server endpoint your Unity client will call to request the dispatch (including example request/response). • Demonstrate handling the CreateDispatch response (agent session id, status) and returning a safe result to the Unity client. Which would you like me to produce next?
b
Since we are using the official LiveKit Node server SDK let's go with option A. We can use the most recent version of that SDK. I'd like server code similar to the example you gave to deploy an agent to my room and would like to be able to specify metadata.
t
Thanks — I’ll show a concise, secure Node.js example using the official LiveKit server SDK to call AgentDispatchService.CreateDispatch, include metadata, and expose a simple Express endpoint your Unity client can call. Before you start • Keep your LIVEKIT_API_KEY and LIVEKIT_API_SECRET on the server only. • Your Unity client should call the server endpoint below (not the LiveKit admin API) to request an agent dispatch. 1) Install dependencies • Install the LiveKit server SDK and express:
Copy code
npm install @livekit/server-sdk express
2) Server code (Express + LiveKit Node server SDK) • This example: ◦ Creates an AgentDispatchService client using the SDK. ◦ Exposes POST /dispatch that accepts JSON { room: string, agent_name: string, metadata?: object }. ◦ Calls CreateDispatch with metadata (encoded as JSON string). ◦ Returns a safe response to the client.
Copy code
// server.js
const express = require('express');
const bodyParser = require('body-parser');
const { AgentDispatchServiceClient } = require('@livekit/server-sdk');
const grpc = require('@grpc/grpc-js');

const app = express();
app.use(bodyParser.json());

// Configure from env
const LIVEKIT_HOST = process.env.LIVEKIT_HOST || 'your-livekit-host:443'; // host[:port]
const LIVEKIT_API_KEY = process.env.LIVEKIT_API_KEY;
const LIVEKIT_API_SECRET = process.env.LIVEKIT_API_SECRET;

if (!LIVEKIT_API_KEY || !LIVEKIT_API_SECRET) {
  console.error('Set LIVEKIT_API_KEY and LIVEKIT_API_SECRET in environment');
  process.exit(1);
}

// Build basic auth metadata for gRPC calls
function authMetadata() {
  const meta = new grpc.Metadata();
  const token = Buffer.from(`${LIVEKIT_API_KEY}:${LIVEKIT_API_SECRET}`).toString('base64');
  meta.set('authorization', `Basic ${token}`);
  return meta;
}

// Create gRPC client for AgentDispatchService
const client = new AgentDispatchServiceClient(
  LIVEKIT_HOST,
  grpc.credentials.createSsl()
);

// POST /dispatch
// Body: { room: string, agent_name: string, metadata?: object }
<http://app.post|app.post>('/dispatch', async (req, res) => {
  try {
    const { room, agent_name, metadata } = req.body;

    if (!room || !agent_name) {
      return res.status(400).json({ error: 'room and agent_name are required' });
    }

    // Build the CreateDispatchRequest using the SDK's request shape.
    // The exact request class helpers vary by SDK version; if the SDK exposes
    // a builder, use it. Here we construct the raw request object expected by gRPC.
    const createDispatchReq = {
      room: room,
      agent_name: agent_name,
      // AgentDispatchRequest metadata field expects a string or Struct depending on proto;
      // encode metadata as JSON string in the `metadata` field so the agent can parse it.
      metadata: metadata ? JSON.stringify(metadata) : undefined,
    };

    // Call CreateDispatch
    const metadataHeaders = authMetadata();
    client.createDispatch(createDispatchReq, metadataHeaders, (err, response) => {
      if (err) {
        console.error('CreateDispatch error:', err);
        return res.status(500).json({ error: err.message || 'dispatch_error' });
      }

      // Response contains dispatch details (agent session id, status, etc).
      // Return relevant safe fields to the Unity client.
      return res.json({
        success: true,
        dispatch: response, // you may filter fields here before returning
      });
    });
  } catch (e) {
    console.error('Dispatch endpoint error:', e);
    res.status(500).json({ error: 'internal_server_error' });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Dispatch server listening on ${PORT}`);
});
3) How metadata is handled • This example puts metadata into the
metadata
field as a JSON string. Agents receiving the dispatch can parse that JSON. • If your agent/LiveKit deployment expects structured metadata differently (e.g., proto Struct), adjust encoding accordingly. The key point: include the data in the CreateDispatch request so the agent gets context (participant id, reason, labels, etc). 4) Security considerations • Authenticate/authorize Unity callers to your /dispatch endpoint (e.g., session cookie, JWT, API token). • Validate incoming requests to prevent misuse of agent dispatching. • Log dispatch outcomes and errors for observability. 5) Return values and follow-up • The createDispatch response includes dispatch/session identifiers and status. Persist any IDs you need for tracking. • You can extend the endpoint to return only minimal info to the client (e.g., { success: true, dispatch_id }) to avoid leaking internals. If you’d like, I can: • Convert the example to use a Promise-based client wrapper (async/await) for cleaner control flow. • Show an example Unity C# call to this /dispatch endpoint (the safe client call). • Tailor the request/response fields returned to your Unity client needs (e.g., return a status polling URL or dispatch id). Which one would you like next?
b
I'd like a C# example for Unity.
t
Thanks — I’ll show a concise Unity C# example that calls your server’s secure /dispatch endpoint (the Node.js endpoint you created) to request an agent be dispatched. This example uses UnityWebRequest, sends JSON with room/agent_name/metadata, and shows basic response handling and error checking. Notes • Do NOT include your LiveKit API key/secret in the Unity client. The client only calls your server endpoint. • Authenticate the Unity call to your server (example shows an Authorization header with a bearer token — adapt to your auth). • Adjust endpoint URL, header names, and JSON fields to match your server. Unity C# example (UnityWebRequest, coroutine)
Copy code
using System;
using System.Collections;
using System.Text;
using UnityEngine;
using UnityEngine.Networking;

[Serializable]
public class DispatchRequest {
    public string room;
    public string agent_name;
    public object metadata; // can be any serializable object (dictionary, anon object)
}

[Serializable]
public class DispatchResponse {
    public bool success;
    public string error;
    // If your server returns dispatch info, map fields here.
    // Example: public string dispatch_id;
    // Or include a generic object for the dispatch payload:
    public object dispatch;
}

public class AgentDispatchClient : MonoBehaviour {
    // Set this to your server endpoint
    [SerializeField] private string dispatchEndpoint = "<https://your-server.example.com/dispatch>";

    // Example call from Unity (coroutine)
    public void RequestAgentDispatch(string room, string agentName, object metadata = null, string bearerToken = null) {
        var req = new DispatchRequest {
            room = room,
            agent_name = agentName,
            metadata = metadata
        };

        StartCoroutine(PostDispatchRequestCoroutine(req, bearerToken));
    }

    private IEnumerator PostDispatchRequestCoroutine(DispatchRequest requestPayload, string bearerToken) {
        string json = JsonUtility.ToJson(requestPayload);

        using (UnityWebRequest www = new UnityWebRequest(dispatchEndpoint, "POST")) {
            byte[] bodyRaw = Encoding.UTF8.GetBytes(json);
            www.uploadHandler = new UploadHandlerRaw(bodyRaw);
            www.downloadHandler = new DownloadHandlerBuffer();
            www.SetRequestHeader("Content-Type", "application/json");

            // If you use bearer token auth from Unity to your server, set it here.
            if (!string.IsNullOrEmpty(bearerToken)) {
                www.SetRequestHeader("Authorization", $"Bearer {bearerToken}");
            }

            yield return www.SendWebRequest();

#if UNITY_2020_1_OR_NEWER
            bool isNetworkError = www.result == UnityWebRequest.Result.ConnectionError;
            bool isHttpError = www.result == UnityWebRequest.Result.ProtocolError;
#else
            bool isNetworkError = www.isNetworkError;
            bool isHttpError = www.isHttpError;
#endif

            if (isNetworkError || isHttpError) {
                Debug.LogError($"Dispatch request failed: {www.error} (status: {www.responseCode})");
                // You can surface this to UI or retry logic
                yield break;
            }

            string respText = www.downloadHandler.text;
            Debug.Log("Dispatch response: " + respText);

            // Parse response (use JsonUtility or a JSON library like Newtonsoft if response is complex)
            try {
                DispatchResponse resp = JsonUtility.FromJson<DispatchResponse>(respText);
                if (resp == null) {
                    Debug.LogWarning("Dispatch response could not be parsed as DispatchResponse.");
                    // Optionally use a JSON library here for dynamic parsing
                } else if (resp.success) {
                    Debug.Log("Agent dispatch requested successfully.");
                    // Handle dispatch info in resp.dispatch if needed
                } else {
                    Debug.LogWarning($"Dispatch failed: {resp.error}");
                }
            } catch (Exception ex) {
                Debug.LogError("Failed to parse dispatch response: " + ex.Message);
            }
        }
    }
}
Usage example • From your UI or code call: ◦ dispatchClient.RequestAgentDispatch("my-room", "my-agent", new { initiatedBy = "unity-client", participantId = "user123" }, bearerToken); Notes on metadata • The example sends
metadata
as an object. Unity's JsonUtility has limits (it doesn't serialize dictionaries). For flexible metadata (dictionaries, nested objects), consider using Newtonsoft.Json (Json.NET for Unity) and replace JsonUtility.ToJson / FromJson with JsonConvert.SerializeObject / DeserializeObject. • Ensure the server decodes the metadata the same way you encoded it (the Node example encodes metadata to JSON string). Error handling & security • Authenticate Unity calls to your server (JWT, session cookies, etc.) so only authorized clients can request agent dispatch. • Implement rate limiting and validation on the server to avoid abuse. • Log the dispatch response on the server; Unity should receive only minimal safe info. If you’d like, I can: • Provide a version using Newtonsoft.Json for flexible metadata/dictionaries. • Show a Unity UI example to collect room/agent name and call the request. • Provide an expected JSON schema for the server response so you can map fields into DispatchResponse. Which would you like next?