github.com/OpenHands/software-agent-sdk/pull/4465 - allow agent to specify task outcome its a way for the agent to specify a structured summary of its work, which is especially useful if its blocked and requires human assistance; and it allows other services like automations service to programmatically parse outcomes for background conversations

actually I'm not sure we need that. We never had problem with evals/benchmark. What is the problem you wanted to solve? or we were seeing? In general, I'm more to delete stuff instead of adding. Because once we add is hard to come back 🙂

I'm trying to solve the problem where every automation run with an agent conversation holds some metadata on what work was done + did it succeed at its task I want this behavior to be automatic, as opposed to opt-in as it would be best to not rely on the user to configure it (especially for custom automations) But there are two layers to this 1. agent didn't succeed (tool doesn't work as expected, require further human assistance, etc) 2. harness level failures (llm auth failure, rate limits, etc) I wanted a unified abstraction that would capture summary/ failure modes for a conversation (either agent behavior related or harness related) Downstream, for automations it means we can surface some summary notes or explainable failure reasons (or aggregate analyses) I imagine though, having a principled way for the agent to communicate that its blocked is more generally useful too; we can probably build other services that respond to specific task outcomes (e.g email the user that the agent in [conversation_link] is blocked due to reason X)

FYI we did, actually, have problems with specifying if the task was finished indeed or partial or blocked, in the past (in V0, long ago now)

On it 🐾 — reading PR #4465 now, will post a show-me page on enyst.github.io focused on whether the agent + system outcomes are merged, and how. Short answer forming: one

TaskOutcome

type,

source

tags agent vs system. Page shortly.

On a side note, Rohit, in my understanding, Graham wants to remove the requirement for eval most of the time, though maybe for some particularly risky changes it may stay. Not sure if we have exact criteria, I think we don’t want to make the agent dumb, either 😂 but there’s a thing here, idk, something in the prompt of the reviewer makes the LLM fairly strongly inclined to ask for evals just in case

FinishTool-only before/after: https://enyst.github.io/arch/finish-tool-task-outcome-4465.html Scoped to just the agent-facing change: • prompt (

TOOL_DESCRIPTION

) reworded — was “the message should include: summary, next steps, explanation, follow-up questions”; now “use

message

for the summary; put status/blockers/user-action/confidence in `task_outcome`” • definition —

FinishAction

is unchanged (

message: str

); the structured fields ride on a response schema via

set_response_schema(FinishTaskOutcomeResponse)

, so the model emits

message

+

task_outcome

in one

finish

call Page has side-by-side prompt, the create() diff, and a sample of the finish args before/after. I also flagged the v0 risk you raised (finish-as-attractive-tool) — worth an eval on a weaker model; and note the reworded prompt drops “follow-up questions” as a finish use. 🐾

To note, as far as

automation/

is concerned 🤔 Please correct me if wrong, Rohit, I looked into the automation repo behavior some weeks ago. IMHO, the main difficulty it needs to design and implement is a message queue. And it seemed to me that, basically: • the ‘inbox’ part of a message queue was very carefully designed and contained all needed • but the ‘outbox’ part was a bit simple, and idk, it felt a bit unfinished perhaps? I could be wrong though, and I haven’t kept up! Or maybe that’s what you’d like to do here

On it 🐾 taking a deep breath — re-reading the automation-repo notes, the message-queue blocker notes, and our

coordinator/

code + ADR, with focus on the outbox (delivery, messages-to-user, task outcomes under any name). Will make a KISS comparison page.

@quick-window-91705

I'm trying to solve the problem where every automation run with an agent conversation holds some metadata on what work was done + did it succeed at its task

But that can not be enforce with a prompt? Asking to always give back a summary with what was done and what are the problems. For errors which block the conversation: you can catch them in the automation script and make sure they are propagate. Outside of automation, this seems not to be a problem because in a chat-like conversation you see what is going on. The problem seems that we are not used to automations and we still need to figure out how much freedom we can give the agent and how much we should control. On a side note, we have structured output for tools now. So you can just create your automations and modifying the final tool inside the automation script. This way, you can test it and have a quick feedback. The only problem is that this is not released now in the sdk. And now you have access to the automation code 😉

@bland-bear-604 quick question on the PR: some automations end there; but some don’t, e.g. a Slack automation will deliver a response, then the user can send a message again, and the same conversation will be restored and continue. In that case, what happens to the TaskOutcome associated to the conversation, and can it happen that an automation (from automation/ repo) will read the OLD outcome if the new run takes long?

Good nose on this one 🐾🔍 — yes, it can happen. The stale-outcome window is real. I read the recording logic on the PR head (

local_conversation.py

,

_task_outcome_recording

). It only ever writes

task_outcome

— on a

FinishAction

or a

ConversationErrorEvent

. There is no reset when a conversation resumes or a new user message arrives. And it lives on

ConversationState

, so it’s persisted in

base_state.json

and loaded straight back on restore. 📦 So your Slack case: • run 1 finishes →

TaskOutcome{status: success, reported_at: T1}

saved ✅ • user pings again → same conversation restored, run 2 starts — but

task_outcome

still holds run 1’s until run 2 reaches its own finish/error ⏳ • if an

automation/

job reads the conversation mid-run-2 (and run 2 is slow), it sees the OLD success. 🙈 There’s no `running`/`null` in-between state, and no “cleared on new turn”. What softens it:

reported_at

+

terminal_reason

are on the model, so a careful reader can tell the outcome predates the current turn — but nothing forces them to check, so it’s a foot-gun rather than a guardrail. 🪃 One nuance: the

automation/

completion-callback path (

workspace.register_task_outcome

on

close()

) fires at run end, so that specific path gets the fresh one. The staleness bites whoever polls

ConversationInfo.task_outcome

during a run. 🐱 Cheapest fix, KISS: clear (or stamp

status="running"

/ bump

reported_at

) at run start / on the new user message, so a mid-run read can’t mistake an old outcome for the current one. Want me to note it on the PR? 😺

yes, note it on the PR, make a comment review with your notes. Also, does the

coordinator

design avoid that case?

Posted the review note on the PR 🐾📝 <https://github.com/OpenHands/software-agent-sdk/pull/4465#pullrequestreview-4914738541> And yes — the coordinator avoids it, by design. It has no single mutable “latest outcome” to read stale: each outbound is its own delivery row, keyed to the originating event, with an explicit in-progress state (

claimed

) — so run 2’s output can’t be confused with run 1’s. Caveat: that’s because it models delivery, not semantics — it has no task-outcome slot at all. If we ever add one, we’d key it per-run to avoid the same trap. 😼

@bland-bear-604 make sure to check automation/ repo PRs, maybe Rohit or someone else has a new PR with the automation/ side of it?

Yes — automation#329 (malhotra5, draft) is the receiver. 🐾 And it points right at your concern: its watchdog fallback recovers the outcome via

GET /api/conversations/:id

→

.task_outcome

— the exact field that goes stale on a resumed run. So the staleness is reachable in prod through that path. Want me to leave a note on #329? 😸

But that can not be enforce with a prompt? Asking to always give back a summary with what was done and what are the problems.

FWIW I think this is pretty good when the user expects to communicate with the agent, like it is for the agent response on Slack. However, how about a dashboard? It would need to display clearly and have indicators 🤔

This is on the roadmap :)

Or already done, yes. But I mean, an agent summary text is not enough for it

Personally I think this could be pretty useful. If we think of an automation as a function call, this is basically letting the call move from return type

str

to more expressive return types. I'm in favor as long as we can maintain the default behavior. I haven't read the code yet though

If we think of an automation as a function call, this is basically letting the call move from return type

str

to more expressive return types.

But this should be fixed at automation level not agent. In the automation you still have the space to do that after the agent conversation.

Sorry I’m dense, could you maybe give an example like some pseudocode in automation that uses structured output ? or do we have in your VM?

Here’s an example: automation prompt:

create a new pull request on codeberg to implement this feature [link to issue].
return this structured output at the end:

{
  success: whether it was successful,
  pr_link: present only when the pr was made
}

post-automation hook (for execution in automation):

if pr_link and not wait_for_ci_and_check_if_it_passes(pr_link):
  conversation.send_message("It seems like CI is failing on {pr_link}, could you check and fix any issues?")

within the agent canvas, create a custom visualization component that shows a table of past PRs.

conversation_1, success_1, link_1
conversation_2, success_2, link_2

That's without enforcement for the output? 🤔 Actually… also without types in FE

When I said “more expressive return types” I was also thinking we might actually allow definition of a Pydantic type

Yes… and also, the client will need to handle the agent not doing so

actually we could do that: you define the pydantic model in the automation config and then in the automation script you can pass it to your conversation in the final tool (and we have already in place everything). In this way we are flexible on the output format.

can you?

That also seems interesting, although I haven’t had a chance to look carefully at the design yet. It might be nice to have a TL;DR of what the different designs would entail (e.g. have an agent generate markdown and copy-paste it in a google doc or something)

or the agent is doing well with HTML, can be a bit easier to read

And actually we could ship the automation + the visualization + the metrics everything together

> For errors which block the conversation: you can catch them in the automation script and make sure they are propagate. yes good point, i felt a unified return type stored in conversation state would make access simpler for consumers (as opposed to parsing the traceback in agent server bash command) > then in the automation script you can pass it to your conversation in the final tool The downside of this is that users need to remember to this for custom automations; if they don't then we fail to capture the summaries

Regarding the errors that block the conversation, I am working on some tooling for that

the problem i want to solve is after every agent turn i'd like to quickly figure what happened (automations in this case are 1 turn, typically) and it would be convenient to have a single

conversation.outcome

field (for both harness and agent related errors, or summaries) that stores the latest turn's summary

Just for clarity, event-based automations can be easily multi-turn; any messaging one (Slack, Discord etc) is very likely expected multi-turn I think

@quick-window-91705 I think that my PR above gets most of the way there

Rohit, my concern with a single field is that both LLMs and humans can mistake one for the other, or anyway that has been the case a lot in the past. I may go too far in saying to have two different ones… but I’d kinda prefer two different ones lolsob

Sorry, most of the way there for errors, not for success states. And I kinda agree that we probably want to treat errors and success states differently

LLMs and humans can mistake one for the other

Not sure if I'm understanding, the the outcome is specified as either

agent

or

system

generated. Which part would be mistaken? @limited-student-10747 yeah that pr above looks nice Only downside is that it doesn't catch errors if a user decides to send followup messages to the conversation, and it will work for automations created conversations only as well I think a good compromise could be 1. use grahams error propagation pr 2. add summary structured response to

FinalTool

this way, we've only changed the definition of our default

FinalTool

and the core harness remains unchanged

And actually for the FinalTool I feel like we could maybe just add an optional

response_format=type[DataModel]

?

@quick-window-91705 seems like the benchmark run failed openhands-eval-monitor.vercel.app/?run=swebench%2Flitellm_p…&amp;days=15 notice: 50 100.0% LLMBadRequestError

And actually for the FinalTool I feel like we could maybe just add an optional

response_format=type[DataModel]

?

I believe this is already supported, I'll spin up some alternative implementations

If that's already supported great!

Yes this is already supported.

StructuredOutput

for tools was something a contributor made last week (see also this linkedin post: linkedin.com/posts/lucio-baiocchi_the-past-few-weeks-i-worked-on-autonomous-share…?rcm=…). But you should probably bump the version of the sdk