Hey folks, I was wondering about the current status of “visual” / image-based input support in the OpenHands SDK. Specifically: is there a supported / stable way for an agent (using the SDK) to autonomously load images from a local directory (or workspace), even when the user’s message does not include any image input, and feed these images into the LLM as visual context/observations? I found issue https://github.com/OpenHands/OpenHands/pull/8552 https://github.com/All-Hands-AI/OpenHands/issues/9944 to be somewhat relevant but they are all OpenHands not the openhands-sdk. Because of this, I’m not sure whether the SDK (or agent runtime) currently guarantees that images can be loaded and processed reliably when they come from local files (not user-uploaded). If there isn’t a stable support yet: would it make sense to treat this as a feature request / enhancement? I’m happy to contribute to enable this functionality.

There is a roundabout way of doing this by using the browser tool. The browser can view things in the file system by changing the path to

file://

. But I think there might be cases where you don’t want a full browser to allow for image support. I think we could add a tool that lets the agent read in an image to the

openhands-tools

directory, and would welcome a contribution!

cool, I'll start working on this and see what happens😇

Actually, SDK's file editor support reading images directly 🙂 https://github.com/OpenHands/software-agent-sdk/blob/bd595d7e5e2d086cbf8ce4f512a33[…]5699a58d1/openhands-tools/openhands/tools/file_editor/editor.py

oh thank you, I completely forgot about file_editor tool here haha.

oh, wow, didn’t realize this 😅

Anyway, I made this tool purely for image input — feel free to ignore it if you think it’s redundant.😇 https://github.com/OpenHands/software-agent-sdk/pull/1306

(sorry about that @flaky-engineer-19286, I should have known this)

this is one of the feature contributed by our community after v1 release, so it wasn't there before🙂

hahaha no problem, I really appreciate it. It's a good opportunity for me to look into the openhands-sdk source code more.