Can you provide working example of setting up a tool for the agent (ie @function_tool) and have the tool return an Image for the model? i’m using a multimodal model but having issues with this process, I can’t yet make this work.

Thanks for reaching out, Pablo. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

somethins like this? @function_tool async def capture_screen_shot(self): “”" Capture a screenshot of the user’s screen and add it to t This allows you to see what the user is currently viewing A nice complement to get_user_activity_context() - you ca Use this tool when: - The user asks you to look at their screen (“can you see - You need to see what the user is working on - The user asks about something visual on their screen - You need to analyze what’s displayed on their monitor - The user asks for help with something they’re viewing Returns: A description of the screenshot capture action. “”" logger.info(“🖥️ Capturing screenshot of user’s screen”) try: # Call the orchestrator API to trigger screenshot cap api_url = “http://localhost:8085/capture-screenshot” with httpx.Client(timeout=10) as client: response = client.post(api_url) if response.status_code == 200: result = response.json() if result.get(“success”) and result.get(“scre logger.info(“🖥️ Screenshot captured succe # The screenshot data should already be b image_data_url = ( f”data:image/png;base64,{result[’scre ) # Add the screenshot to the chat context try: # Get agent session from singleton session = self.get_session() if session and hasattr(session, “ctx” # Add the image to the chat conte session.ctx.add_message( role=“user”, content=[ “Here is my current scree ImageContent(image=image_ ], ) logger.info(“🖥️ Screenshot image return “I’ve captured a screensho else: logger.warning( “⚠️ Could not access session ) # Fallback: return string descrip return “I’ve captured a screensho except Exception as context_error: logger.error( f”❌ Error adding screenshot to c ) # Fallback: return string description return “I’ve captured a screenshot of else: error_msg = result.get(“error”, “Unknown logger.error(f”❌ Screenshot capture fail return f”Screenshot capture failed: {erro else: logger.error( f”❌ Screenshot API request failed with s ) return f”Screenshot capture failed: API reque except Exception as e: logger.error(f”❌ Error capturing screenshot: {e}“) return f”Error capturing screenshot: {str(e)}”

Here’s a friendly guide to help you set up a function tool for capturing a screenshot in LiveKit: 1. Define the Tool: Start by using the

@function_tool

decorator to create a tool that handles taking screenshots. This tool will manage the process of triggering the screenshot and returning the result.

from livekit.agents import function_tool, Agent, RunContext

class MyAgent(Agent):
    @function_tool()
    async def capture_screenshot(self, context: RunContext, name: str = "screenshot_1") -> dict:
        """Capture a screenshot of the screen."""
        try:
            # Platform-specific implementation here
            screenshot = capture_screen()
            return {"screenshot": screenshot}
        except Exception as e:
            return {"error": str(e)}

2. Platform-Specific Implementation: You’ll want to tailor the screenshot functionality depending on the platform you’re working on: ◦ iOS: Use

RPScreenRecorder

for capturing the screen within your app. ◦ Android: Use

MediaProjectionManager

to ask for permission to capture the screen. ◦ macOS: Use

ScreenCaptureKit

to record the screen. ◦ Unity: Use

ScreenCapture.CaptureScreenshotIntoRenderTexture

to grab screenshots. 1. Register the Tool: Finally, add your tool to the LiveKit Agent and make sure it works smoothly with the chat context so the captured screenshot shows up nicely. If you want me to walk you through any part in more detail or need examples tailored to your platform, just let me know—I’m here to help!

yes, how to i make sure it works smoothly with the chat context? what exactly is supposed to be returned? you are returning a dictionary, why?

Hey there! To make sure the screenshot capture tool works smoothly with the chat context, here’s a simple step-by-step guide: 1. Create an ImageContent Object: Take the screenshot data you’ve captured and turn it into an

ImageContent

object. You can initialize it with a base64 data URL, an external URL, or a

VideoFrame

.

import base64
from livekit.agents.llm import ImageContent

# Convert image bytes to base64
base64_image = f"data:image/png;base64,{base64.b64encode(image_bytes).decode('utf-8')}"

# Create ImageContent
chat_image = ImageContent(image=base64_image)

2. Add ImageContent to ChatContext: Next, use the

add_message

method to add the image into the chat context.

from livekit.agents.llm import ChatContext

chat_ctx = ChatContext()
chat_ctx.add_message(
    role="user",
    content=[chat_image]
)

3. Return Value: Your tool should send back a confirmation message or the image data itself. Returning a dictionary is optional but can be handy if you want to include extra info like success status, image URL, or the base64 string for debugging or further steps. By adding the

ImageContent

to the chat context, the LLM can easily work with the image alongside any text. If anything’s unclear or you want to chat more about this, just let me know—I’m here to help! Sources: tests/test_chat_ctx.py

how to i build the chat context, when the agent decided when to call the tool?

Here’s a friendly step-by-step on how to build a chat context when the agent decides to call a tool: 1. Initiate the Tool Call: The agent realizes it needs a tool and triggers the function call. This creates a

FunctionCall

object with details like the tool's name and arguments. 2. Execute the Tool: The tool runs asynchronously, and the agent’s state switches to "thinking" to show it’s processing. 3. Handle the Tool Output: Once the tool finishes, it produces a

FunctionCallOutput

with the result or an error. This gets added to the chat context. 4. Update the Chat Context: Add both the

FunctionCall

and

FunctionCallOutput

to the chat context. This keeps the conversation history complete with the tool interaction. 5. Resume the Conversation: Use the updated chat context to generate the next response, so the conversation flows smoothly. For example, if the agent calls a tool to fetch weather data, the chat context might look like this:

{
  "items": [
    {
      "type": "function_call",
      "name": "get_weather",
      "arguments": "{\"location\": \"Tokyo\"}"
    },
    {
      "type": "function_call_output",
      "output": "The weather in Tokyo is sunny today."
    }
  ]
}

This way, the conversation stays clear and informed by the tool’s output. If you want me to explain anything more, just let me know—I’m here to help!

i want a complete example! i don’t get this. here’s my tool: @function_tool async def capture_screen_shot(self): “”" Capture a screenshot of the user’s screen and add it to t This allows you to see what the user is currently viewing A nice complement to get_user_activity_context() - you ca Use this tool when: - The user asks you to look at their screen (“can you see - You need to see what the user is working on - The user asks about something visual on their screen - You need to analyze what’s displayed on their monitor - The user asks for help with something they’re viewing Returns: A description of the screenshot capture action. “”" logger.info(“🖥️ Capturing screenshot of user’s screen”) try: # Call the orchestrator API to trigger screenshot cap api_url = “http://localhost:8085/capture-screenshot” with httpx.Client(timeout=10) as client: response = client.post(api_url) if response.status_code == 200: result = response.json() if result.get(“success”) and result.get(“scre logger.info(“🖥️ Screenshot captured succe # The screenshot data should already be b image_data_url = ( f”data:image/png;base64,{result[’scre ) # Add the screenshot to the chat context try: # Get agent session from singleton session = self.get_session() if session and hasattr(session, “ctx” # Add the image to the chat conte session.ctx.add_message( role=“user”, content=[ “Here is my current scree ImageContent(image=image_ ], ) logger.info(“🖥️ Screenshot image return “I’ve captured a screensho else: logger.warning( “⚠️ Could not access session ) # Fallback: return string descrip return “I’ve captured a screensho except Exception as context_error: logger.error( f”❌ Error adding screenshot to c ) # Fallback: return string description return “I’ve captured a screenshot of else: error_msg = result.get(“error”, “Unknown logger.error(f”❌ Screenshot capture fail return f”Screenshot capture failed: {erro else: logger.error( f”❌ Screenshot API request failed with s ) return f”Screenshot capture failed: API reque except Exception as e: logger.error(f”❌ Error capturing screenshot: {e}“) return f”Error capturing screenshot: {str(e)}” tell me exactly how I can make my agent use the screenshot returned by this tool.

Here’s a complete example of how to integrate your screenshot tool with the chat context in LiveKit: 1. Define the Tool: You’ve got your screenshot tool ready! Just make sure it returns the screenshot as a base64-encoded string or a URL. 2. Update the Chat Context: We’ll use the

ImageContent

class to add that screenshot right into the chat context. Here’s a friendly tweak to your tool to make that happen:

isn’t your example truncated?

You're absolutely right, the example might look a bit cut off because of formatting. No worries—here’s the full, clear example to help you integrate your screenshot tool with the chat context: 1. Define the Tool: Make sure your tool captures the screenshot and returns it as a base64-encoded string or a URL. You can tweak your tool like this: