i want to make a tool that when called adds an image to the conversation, so that the agent can describe it. how can i do this?

Thanks for reaching out, Antonio. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

what if i need to declare this tool outside of the agent?

You're asking how to declare the image tool outside the Agent class. You can do that and then register or attach it to one or more agents. Below are concise examples and the key options. Python (module-level tool)

from livekit.agents import function_tool, Agent, RunContext, ImageContent

@function_tool()
async def add_image_to_conversation(
    context: RunContext,
    image_url: str,
    description: str = ""
):
    image = ImageContent(image=image_url)
    context.chat_ctx.add_message(role="user", content=[description or "User uploaded an image.", image])
    return f"Image from {image_url} added to conversation."

Register with an agent at construction:

class MyAgent(Agent):
    def __init__(self):
        super().__init__(instructions="You analyze images.", tools=[add_image_to_conversation])

Or add to an existing agent instance dynamically:

await agent.update_tools(agent.tools + [add_image_to_conversation])

Node.js (module-level tool)

import { llm } from '@livekit/agents';
import { z } from 'zod';

export const addImageToConversation = llm.tool({
  description: 'Add image to conversation',
  parameters: z.object({ imageUrl: z.string().url(), description: z.string().optional() }),
  execute: async ({ imageUrl, description }, { ctx }) => {
    ctx.chatCtx.addMessage({
      role: 'user',
      content: [{ type: 'text', text: description || 'User uploaded an image.' },
                { type: 'image_url', image_url: { url: imageUrl } }]
    });
    return `Image from ${imageUrl} added to conversation.`;
  },
});

Attach to an agent:

class MyAgent extends voice.Agent {
  constructor() {
    super({ instructions: 'You analyze images.', tools: { addImageToConversation } });
  }
}

Key points • External tools have access to

RunContext

/

ctx.chatCtx

so they can add messages directly. • You can reuse the tool across agents by passing it in

tools

or updating an agent's tools dynamically. • If you want the agent to describe immediately, the tool can also trigger

generate_reply

(Python) or call session methods after adding the image. Which language (Python or Node) and whether you want the agent to auto-describe the image or wait for a prompt? I’ll provide a tailored snippet. Sources: Tool definition and use | LiveKit docs

context.chat_ctx does not exists

Thanks — you're right to flag that. Use

context.session.chat_ctx

(not

context.chat_ctx

) inside a tool, then persist changes with

await context.session.update_chat_ctx(...)

. Quick corrected example (module-level tool):

from livekit.agents import function_tool, ImageContent, RunContext

@function_tool()
async def add_image_to_conversation(context: RunContext, image_url: str, description: str = ""):
    chat_ctx = context.session.chat_ctx
    chat_ctx.add_message(
        role="user",
        content=[description or "User uploaded an image.", ImageContent(image=image_url)]
    )
    await context.session.update_chat_ctx(chat_ctx)
    # optional: trigger an immediate description
    await context.session.generate_reply(instructions=f"Describe the image at {image_url}")
    return f"Image from {image_url} added to conversation."

If you want this tool used by an agent, pass it in

tools

when constructing the agent or call

await agent.update_tools(...)

to add it dynamically. Do you want the agent to auto-describe the image (use

generate_reply

) or wait for an explicit prompt? Also tell me which language/runtime you're using if you want a tailored snippet. Sources: Tool definition and use | LiveKit docs | Pipeline nodes and hooks | LiveKit docs | Vision | LiveKit Docs

also, update_chat_ctx does not exists

You're right to call that out — persist the changed chat context via the agent or session, not on the RunContext itself. Correct pattern (Python tool defined outside an agent):

from livekit.agents import function_tool, ImageContent, RunContext

@function_tool()
async def add_image_to_conversation(context: RunContext, image_url: str, description: str = ""):
    # get a copy of the current chat context from the agent or session
    chat_ctx = context.agent.chat_ctx.copy()  # or: context.session.chat_ctx.copy()

    # add the image message
    chat_ctx.add_message(
        role="user",
        content=[description or "User uploaded an image.", ImageContent(image=image_url)]
    )

    # persist the updated context via the agent (or session)
    await context.agent.update_chat_ctx(chat_ctx)
    # await context.session.update_chat_ctx(chat_ctx)  # alternative

    # optional: trigger immediate description
    await context.session.generate_reply(instructions=f"Describe the image at {image_url}")

    return f"Image from {image_url} added to conversation."

Notes: • Use

context.agent.chat_ctx

or

context.session.chat_ctx

(not

context.chat_ctx

). • Call

await context.agent.update_chat_ctx(...)

or

await context.session.update_chat_ctx(...)

to persist changes —

update_chat_ctx

exists on the agent/session, not on the RunContext itself. • Use

ImageContent

for the image payload; if you need to download/validate the image first, use the images plugin before constructing

ImageContent

. Do you want the tool to auto-describe images, or should it just add them and wait for a separate prompt? Also tell me if you want the Node.js variant. Sources: Tool definition and use | LiveKit docs | Agents and handoffs | LiveKit docs

can i simply return the image as a result?

Short answer: No — returning the image data from a tool just sends a string back as the tool result. To make the LLM/agent actually see and describe the image, the tool must add an ImageContent message into the chat context and persist that context. Correct pattern (concise): • Create an

ImageContent

with the image (URL or base64). • Add a user message containing that

ImageContent

to the chat context obtained from

context.session.chat_ctx

or

context.agent.chat_ctx

. • Persist the updated context with

await context.session.update_chat_ctx(chat_ctx)

or

await context.agent.update_chat_ctx(chat_ctx)

. • Optionally call

await context.session.generate_reply(...)

to have the agent describe it immediately. If you want, tell me: • which language (Python or Node.js) and • whether you want the agent to auto-describe the image or wait for a prompt, and I’ll provide a minimal exact snippet for your setup. Sources: Tool definition and use | LiveKit docs | External data and RAG | LiveKit docs | Vision | LiveKit docs

using update_chat_ctx i get this error: "cannot access local variable 'conversation_item' where it is not associated with a value"

Thanks — I can help debug that error. That runtime message ("cannot access local variable 'conversation_item' where it is not associated with a value") usually means something in the chat-context update flow is missing or malformed. To diagnose, I need a little more info: 1. Which runtime / language are you using? (Python or Node.js) and LiveKit Agents version. 2. Paste the exact tool code you call (the function body) and the exact call that triggers the error. 3. Are you modifying the chat context by copying it (e.g.,

chat_ctx = context.session.chat_ctx.copy()

), then calling

await context.session.update_chat_ctx(chat_ctx)

(or

context.agent.update_chat_ctx

)? Include that snippet. 4. If you can, include the full stack trace or the surrounding error lines. While you gather that, check these quick points: • Ensure you call

.copy()

on the chat context before mutating it. • Each message added must be a valid ChatMessage / content entry (text or ImageContent). Avoid passing incomplete objects or None in the

content

list. • If you build a message dict manually, ensure all required fields (type/text or image payload) are present. • If you use plugins to download/convert images, confirm the returned image data is non-empty before wrapping in

ImageContent

. Send the code and stack trace and I’ll point out the exact problematic line and a corrected snippet.

Traceback (most recent call last): File \"c:\\Projects\\.venv\\Lib\\site-packages\\livekit\\agents\\voice\\generation.py\", line 591, in _traceable_fnc_tool val = await function_callable() ^^^^^^^^^^^^^^^^^^^^^^^^^ File \"C:\\Projects\\src\\models\\features\\file_publisher_feature.py\", line 223, in retrieve_document_content await ctx.session.current_agent.update_chat_ctx(chat_ctx) File \"c:\\Projects\\.venv\\Lib\\site-packages\\livekit\\agents\\voice\\agent.py\", line 206, in update_chat_ctx await self._activity.update_chat_ctx( File \"c:\\Projects\\.venv\\Lib\\site-packages\\livekit\\agents\\voice\\agent_activity.py\", line 342, in update_chat_ctx await self._rt_session.update_chat_ctx(chat_ctx) File \"c:\\Projects\\.venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 1106, in update_chat_ctx events = self._create_update_chat_ctx_events(chat_ctx) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"c:\\Projects\\.venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 1168, in _create_update_chat_ctx_events _create_item(previous_msg_id, msg_id) File \"c:\\Projects\\.venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 1147, in _create_item item=livekit_item_to_openai_item(chat_item), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"c:\\Projects\\.venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\utils.py\", line 223, in livekit_item_to_openai_item conversation_item.id = item.id ^^^^^^^^^^^^^^^^^ UnboundLocalError: cannot access local variable 'conversation_item' where it is not associated with a value

Thanks — that traceback is helpful. The UnboundLocalError occurs while converting your LiveKit ChatItem into the OpenAI realtime format; it means a variable used when building a conversation item was never set, which usually happens when a chat message you added is missing required fields or has an unexpected shape. What I need to debug this with you: 1. The exact code you use to build the message(s) you add to

chat_ctx

(the snippet that calls

chat_ctx.add_message(...)

or constructs message dicts). 2. The minimal snippet that copies and updates the chat context (how you obtain

chat_ctx

, call

.copy()

, then call

await context.session.update_chat_ctx(...)

or

await context.agent.update_chat_ctx(...)

). 3. Which LiveKit packages and versions you have installed (pip freeze lines for livekit packages or version numbers). Quick checks you can run now (most common causes): • Ensure you call

chat_ctx = context.session.chat_ctx.copy()

(or

context.agent.chat_ctx.copy()

) before mutating. • When adding a message, pass valid content items. Use high-level helpers instead of raw dicts where possible. Example valid content forms: ◦ Plain text: content=["some text"] ◦ ImageContent: content=[ImageContent(image=image_url)] ◦ Mixed: content=["desc", ImageContent(image=image_url)] • Don’t append None or partially-built dicts into

content

. Every content entry must include the necessary fields (text or image payload). • If you construct ChatMessage or content as dicts, include

id

,

role

, and content entries in the correct format — better to call

chat_ctx.add_message(role="user", content=[...])

which sets required fields. • If you download/transform the image, confirm the result is non-empty and valid before wrapping in

ImageContent

. If you paste the small code snippets I asked for, I’ll point out the exact problematic line and provide a corrected snippet you can drop in.

this is my current implementation chat_ctx = ctx.session.history.copy() # Add the message to the chat context chat_ctx.add_message( role="user", content=[ImageContent(image=file_url)] ) # Update the chat context await ctx.session.current_agent.update_chat_ctx(chat_ctx)

Thanks — that snippet explains the error. Two issues to fix: 1) Use the session/agent chat context object names correctly •

ctx.session.history

is not the right chat-context object to copy. Use

ctx.session.chat_ctx

or

ctx.agent.chat_ctx

(or

ctx.session.current_agent.chat_ctx

if that’s how you access the agent). Copy that before mutating. 2) Ensure you persist via the same object that owns update_chat_ctx • Call

await ctx.session.update_chat_ctx(...)

or

await ctx.session.current_agent.update_chat_ctx(...)

— whichever is the owner in your runtime. Don’t mix

history

with

chat_ctx

when creating events for the OpenAI realtime plugin. Minimal corrected pattern (drop-in):

# get a proper copy of the chat context
chat_ctx = ctx.session.chat_ctx.copy()

# add the image message (ImageContent must be a valid object)
chat_ctx.add_message(
    role="user",
    content=[ImageContent(image=file_url)]
)

# persist via the session (or agent) that owns update_chat_ctx
await ctx.session.update_chat_ctx(chat_ctx)

Why this fixes the UnboundLocalError • The OpenAI realtime conversion expects well-formed ChatItems from

chat_ctx

. Using the correct

chat_ctx

and persisting it with the matching

update_chat_ctx

ensures every message has the required fields so the conversion code can create conversation items without hitting an uninitialized local variable. If that still errors, please paste: • The exact object you get when you print

type(ctx.session.chat_ctx)

and

repr(chat_ctx)

after copying, and • The LiveKit packages/versions you have (pip freeze lines for livekit packages). I’ll inspect the object shape and point out any missing fields.