Is it possible to test by feeding image to the agent to test it's multimodality? • https://docs.livekit.io/agents/build/testing/

Thanks for reaching out, Galih. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Can you give me sample code? just example to test including image on the chat context?

You're asking for a sample showing how to include an image in the chat context to test multimodality. Below are two concise examples: one that constructs a

ChatContext

with an

ImageContent

at initialization, and one that shows uploading/sending image data dynamically. Replace placeholders (API keys, agent IDs, image URLs/base64) with your values. 1) Initialization — include an external image URL or base64 in `ImageContent`:

// Node.js / JavaScript example (pseudo-API shape)
import { AgentsClient, ChatContext, ChatMessage, ImageContent } from 'livekit-agents-sdk';

const client = new AgentsClient({ apiKey: 'YOUR_API_KEY' });

async function testWithImageInit() {
  const image = new ImageContent({
    // Use either a remote URL:
    url: '<https://example.com/test-image.jpg>',
    // Or a base64 data URL:
    // dataUrl: 'data:image/png;base64,iVBORw0K...'
    mimeType: 'image/jpeg',
  });

  const ctx = new ChatContext({
    messages: [
      new ChatMessage({
        role: 'user',
        content: [
          { type: 'text', text: "What's in this image?" },
          image,
        ],
      }),
    ],
  });

  const resp = await client.runAgent({
    agentId: 'YOUR_AGENT_ID',
    context: ctx,
    // optional: specify model/plugin that supports vision
    model: 'openai-vision-model',
  });

  console.log('Agent response:', resp);
}

testWithImageInit().catch(console.error);

2) Dynamic upload — send image bytes from a frontend or buffer during a session:

// Node.js / JavaScript example (pseudo-API shape)
import fs from 'fs';
import { AgentsClient, ChatContext, ChatMessage, ImageContent } from 'livekit-agents-sdk';

const client = new AgentsClient({ apiKey: 'YOUR_API_KEY' });

async function testWithImageUpload() {
  // Read image bytes and convert to data URL
  const buffer = fs.readFileSync('./local-image.png');
  const b64 = buffer.toString('base64');
  const dataUrl = `data:image/png;base64,${b64}`;

  const image = new ImageContent({
    dataUrl,
    mimeType: 'image/png',
  });

  const ctx = new ChatContext({
    messages: [
      new ChatMessage({
        role: 'user',
        content: [
          { type: 'text', text: "Please describe this image." },
          image,
        ],
      }),
    ],
  });

  const resp = await client.runAgent({
    agentId: 'YOUR_AGENT_ID',
    context: ctx,
    model: 'openai-vision-model',
  });

  console.log('Agent response:', resp);
}

testWithImageUpload().catch(console.error);

Notes: • Ensure the agent is configured with a vision-capable LLM/plugin (e.g., OpenAI/Gemini vision) so it can interpret image content. • Use

mimeType

that matches the image format. • For large images prefer remote URLs or streaming rather than embedding huge base64 strings. Want the same examples in Python or a frontend (browser) flavor? Which model/plugin are you testing with?

Give me python example

You're asking for a Python example that includes an image in the chat context to test a multimodal agent. Below are two concise Python examples: one that constructs the chat context with an external image URL or base64 data URL at initialization, and one that embeds a local image as a base64 data URL (dynamic upload). Replace placeholders (API key, agent ID, model) with your values. 1) Initialization — include an external image URL or base64 in ImageContent:

# Python (pseudo-API shape)
from livekit_agents import AgentsClient, ChatContext, ChatMessage, ImageContent

client = AgentsClient(api_key="YOUR_API_KEY")

def test_with_image_init():
    image = ImageContent(
        url="<https://example.com/test-image.jpg>",
        mime_type="image/jpeg",
    )

    msg = ChatMessage(
        role="user",
        content=[
            {"type": "text", "text": "What's in this image?"},
            image,
        ],
    )

    ctx = ChatContext(messages=[msg])

    resp = client.run_agent(
        agent_id="YOUR_AGENT_ID",
        context=ctx,
        model="openai-vision-model",  # use a vision-capable model/plugin
    )

    print("Agent response:", resp)

if __name__ == "__main__":
    test_with_image_init()

2) Dynamic upload — embed a local image as a base64 data URL:

# Python (pseudo-API shape)
import base64
from livekit_agents import AgentsClient, ChatContext, ChatMessage, ImageContent

client = AgentsClient(api_key="YOUR_API_KEY")

def test_with_image_upload():
    with open("local-image.png", "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    data_url = f"data:image/png;base64,{b64}"

    image = ImageContent(
        data_url=data_url,
        mime_type="image/png",
    )

    msg = ChatMessage(
        role="user",
        content=[
            {"type": "text", "text": "Please describe this image."},
            image,
        ],
    )

    ctx = ChatContext(messages=[msg])

    resp = client.run_agent(
        agent_id="YOUR_AGENT_ID",
        context=ctx,
        model="openai-vision-model",
    )

    print("Agent response:", resp)

if __name__ == "__main__":
    test_with_image_upload()

Notes: • Ensure your agent uses a vision-capable LLM/plugin (e.g., OpenAI/Gemini vision). • Use the correct

mime_type

and reasonable image sizes (remote URLs or streaming are preferable for large files). Would you like this adapted for a specific SDK version or a real example using a particular model/plugin?

give me example on a test file like this:

import pytest
from livekit.agents import AgentSession
from livekit.plugins import openai
from dapagent import Assistant

@pytest.mark.asyncio
async def test_assistant_response() -> None:
    """
    Test that the agent responds helpfully to a user greeting.
    """
    async with (
        openai.LLM(model="gpt-4o-mini") as llm,
        AgentSession(llm=llm) as session,
    ):
        await session.start(Assistant())

        result = await session.run(user_input="Hello")

        await result.expect.next_event().is_message(role="assistant").judge(
            llm, intent="Makes a friendly introduction and offers assistance."
        )

You're asking for a pytest test file like your example that includes an image in the chat context. Below is a concise async pytest example that mirrors your style, uses the openai plugin, and adds an

ImageContent

to the chat message so the agent can be tested for multimodal responses. Replace placeholders (agent class, model, image path/URL, and assertion intent) with your values.

import base64
import pytest
from livekit.agents import AgentSession, ChatMessage, ImageContent, ChatContext
from livekit.plugins import openai
from dapagent import Assistant  # your agent implementation

@pytest.mark.asyncio
async def test_assistant_handles_image() -> None:
    """
    Test that the agent can process an image included in the chat context
    and respond appropriately.
    """
    # Prepare image as data URL (alternatively use external URL in ImageContent)
    with open("tests/fixtures/test-image.png", "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    data_url = f"data:image/png;base64,{b64}"

    image_content = ImageContent(
        data_url=data_url,
        mime_type="image/png",
    )

    user_msg = ChatMessage(
        role="user",
        content=[
            {"type": "text", "text": "Please describe what's shown in this image."},
            image_content,
        ],
    )

    ctx = ChatContext(messages=[user_msg])

    async with (
        openai.LLM(model="gpt-4o-mini") as llm,
        AgentSession(llm=llm, context=ctx) as session,
    ):
        await session.start(Assistant())

        result = await session.run(user_input=None)  # context already contains the user message

        # Expect the next event to be an assistant message that demonstrates understanding of image
        await result.expect.next_event().is_message(role="assistant").judge(
            llm, intent="Describes the contents of the provided image and offers assistance."
        )

If you prefer using an external image URL instead of embedding base64, replace the

ImageContent

creation with:

image_content = ImageContent(url="<https://example.com/test-image.jpg>", mime_type="image/jpeg")

Want this adapted for a different model/plugin or to test multiple image cases?