<@U06PEF04LEQ> <@U09DMU3EM2T> <@U06PYTVAES1> Hitt...
# proj-agent
c
@limited-student-10747 @strong-ice-84145 @calm-artist-7206 Hitting this issue with a customer. One way to solve it is at the integration layer but I think if we do that we'll have to solve it over and over with every integration and this won't catch all of the places this can happen in MCPs... What do you think about the proposed solution here? github.com/OpenHands/software-agent-sdk/issues/4111
l
Interesting, I understand the problem, let me see if I can gather information about how other tools solve this
w
There's a little checkbox to upload as a file instead of a multimodal image!
c
That doesn't help us with images obtained by integrations though.
w
Screenshot 2026-07-14 at 9.19.17 AM.png
l
This is specifically for MCP responses I think
w
ohhh
fair
what if we just always saved the image in
/tmp/
, and told the agent about it? That would solve the UX problem above too--it's a little hard to discover
c
Yep -- that is my thinking too. I looked at doing this at LLM layer but that has some downsides -- like that it would fire every turn, not just the first time. But if we provide tool and inject instruction when the image is provided, seems like that would do it.
l
@nutritious-balloon-65098 find out what opencode, pi, and codex do in the case of image responses from MCP servers, do they save it to disk and tell the agent, or just append it as multimodal content in a message?
n
I'm on it! Graham Neubig can track my progress here.
I’ll trace this from source where possible rather than relying on docs. I’m going to identify the relevant repositories first, then inspect their MCP result handling paths for image content.
rg
is not installed in this container, so I’m falling back to `find`/`grep`. The workspace also didn’t show a Git remote from that first probe, so I’ll inspect the project layout before deciding whether local code is relevant.I found current public repos for all three:
anomalyco/opencode
,
earendil-works/pi
, and
openai/codex
. I’m going to clone shallow copies under
/tmp
and inspect the MCP tool-result conversion code, since docs/issues only tell part of the story.The repos are available locally now. I’m narrowing each one to code paths that mention MCP result content types such as
image
,
mimeType
,
data
, and model-message conversion.A quick read already shows Pi’s core transcript type supports
ImageContent
directly in tool results, so Pi is likely inline unless an extension changes it. I’m checking the actual MCP adapter/package path now, because Pi’s base repo may not ship MCP in the same way as the other two.For OpenCode, the key question is how
client.callTool()
output is returned into the LLM tool-result. I found that adapter path and am reading around the MCP execute function rather than the setup/status code.OpenCode does turn MCP image blocks into
attachments
with
data:
URLs, and also keeps the original MCP
content
on the tool result object. I’m checking how those attachments are lowered into model input, because that determines whether the agent sees them as multimodal content or only as UI/session metadata.The OpenCode path is more nuanced than “just append”: it stores the image in session state as an attachment with a data URL, then either sends it inside the tool-result content if the provider supports that, or extracts it into a separate user message when the provider does not. It still is not written to disk.That synthetic user-message fallback has the exact behavior we care about, so I’m reading the following lines and the synthetic prompt text to state it accurately.OpenCode is now mostly clear. Next I’m checking Pi’s official repo for whether MCP is built-in or supplied through extension packages, then I’ll inspect the common Pi MCP extension behavior if needed.Pi’s official coding agent explicitly says it has no built-in MCP. Its core tool-result model does carry image blocks inline, and unsupported models get text placeholders rather than files. I’m going to inspect the commonly referenced
pi-mcp-adapter
package too, since that is likely what people mean by “Pi + MCP” in practice.For Pi, provider adapters matter. Anthropic can pass image blocks directly; OpenAI-compatible chat APIs appear to add image blocks after tool results in a follow-up user message. I’m reading the surrounding code so I don’t overstate that.I cloned two Pi MCP extension repos that show up in current search results. I’ll use them only to qualify Pi’s “MCP behavior through extensions,” because they are not the built-in Pi core.The two Pi extensions differ: the lightweight older
pi-mcp-tools
drops images to a text placeholder, while
pi-mcp-adapter
explicitly passes image blocks through. I’m going to use
pi-mcp-adapter
as the current/main evidence and mention the older extension as a caveat only if useful.Now I’m on Codex. I’m following the in-process MCP tool handler and the Responses payload builder, because Codex has both UI rendering and model-context paths.Codex executes the MCP call and wraps the returned
CallToolResult
as a `McpToolOutput`; the next part is how
McpToolOutput
serializes itself into the model-facing function-call output.Codex converts MCP
{type:"image"}
blocks into
input_image
items in the function-call output payload when the result has no
structuredContent
. One important caveat I found: if
structuredContent
is present, Codex serializes that instead and does not use the MCP
content[]
blocks for model context.Codex also has a history-management path that can replace old tool-output images with text placeholders when trimming history, but the initial MCP result path is still multimodal payload, not “save to disk and tell the agent.”I checked current source as of July 14, 2026. Bottom line: none of these broadly do the “always save image to
/tmp
and tell the agent the path” approach for MCP image responses. The common pattern is inline multimodal content, with fallbacks/placeholders for unsupported models and some temp-file handling only for oversized text/raw details, not the images themselves., with interactive elements
c
So there is a logical place to intercept for MCP in the code, but the problem is that only handles MCP responses. It doesn't handle the other integration paths or the UI path. Instead of fixing in at least 3 places, plus a new one for every new integration -- we can build on some code exposed by VisionInspectTool we can solve this in one place that is future proof.
c
The agent in the linked issue talks about “the LLM layer”, but that’s late; couldn’t it be somewhere at ~conversation layer, such as, an
on_event
, or where we already check the latest user message
nod 1
If the agent has a tool that converts the latest image, what if it gets two images, or if it forgets and continues and then we want it?
c
Digging deeper it also looks like we silently discard other types of multipart content making them irretrievable later.
• MCP (
mcp/definition.py:66
): only
TextContent
and
ImageContent
blocks are kept. Any other block type (embedded resources, audio, blobs) hits
else: logger.warning("Unsupported ... Ignoring")
— dropped.
s
find out what opencode, pi, and codex do in the case of
Wow this is my new favorite prompt amaze can't wait to use it in any (non-applicable) circonstance to solve any minor issue 😛
😂 2
c
Oh that’s interesting: audio really seems like a good case. My agent uses audio… but with files, not audio! So far, what it does is: • receive audio files saved at a location it knows -> use a SST tool on them • write a thing -> use a TTS tool on it -> put the audio file in outbox -> off it goes (by ‘tool’ here I mean some application, a cli, like whisper)
nod 1
c
Okay. I think I'll redraft the issue description to 1. Be more generic -- not just image data 2. Suggest a callback in the
_on_event
chain
👍 1
c
@limited-student-10747 Maybe with Vasco’s prompt pieces refactoring, we can have a section where we tell the agent from what channel is the request coming. e.g. “Environment information: you have received this user request from Slack. Make sure you format your responses properly for display in Slack.”
c
That won't fix the binary/other parts dropped in
mcp/definition.py
but it should allow us to handle that in a separate issue. Drafting new issue desc now...
🙏 1