https://docs.all-hands.dev/ logo
Join Slack
Powered by
Hey PR to try to fix stuck detecion for GLM model...
# proj-agent
m
Hey PR to try to fix stuck detecion for GLM models: github.com/OpenHands/software-agent-sdk/pull/4332 This problem was making more than a couple of automations failing. Reviews/feedbacks welcome šŸ™‚
c
I see the usefulness of extending it, and telling the LLM even more clearly about it
I'm not sure I fully understand though at a glance, what happens if it continues: I think we do need a terminal STUCK state, where the conversation gives up before max iterations, and returns to the user
I understand that for Automations that isnā€™t idealā€¦ but maybe an Automation could extend its lifetime of tries withā€¦ idk, a hook? a higher value?
Do you by any chance have a trajectory where we can see the exact context sent to the LLM and its repeated responses?
m
conversation_4cbe4fea-66f9-4e31-8767-4c25fc7cb9c0.zip
conversation_4cbe4fea-66f9-4e31-8767-4c25fc7cb9c0.zip
this is one example
But let me be precise about what changed vs. what didn't. On the terminal state: it's still there, unchanged. 
STUCK
is still a hard-terminal 
execution_status
, same tier as `FINISHED`/`PAUSED`/`ERROR` the run loop breaks out of while True the moment it's set, before calling 
agent.step()
again, and control returns to the caller. This PR doesn't touch that mechanism at all. All it does is move the judgment one repeat later and add an explicit message in between: To be even more precise: - Before: 3 identical action+error pairs, is_stuck() fires immediately, 
STUCK
. - Now: 3 identical, one corrective nudge is injected ("You've called X the same way 3 times and gotten the same error..."), loop continues. If the model repeats the exact same call a 4th time, 
STUCK
, same as before, no extra iterations burned beyond that one nudge. So it is not keep retrying but give one last chanche before stopping it. If it ignores the nudge, it's stuck exactly as terminally as it was before this PR. Or what do you think about that?
šŸ‘ 1
c
This is interestingā€¦
Copy code
{
  "id": "df416fe5-d5eb-4b96-93c5-72be37e920cb",
  "timestamp": "2026-08-01T18:34:00.366134",
  "source": "environment",
  "parent_id": "11da6fc7-560a-43fd-9df0-421ce3dced6b",
  "tool_name": "terminal",
  "tool_call_id": "b5ab3b22-cea8-4b00-b5e8-668f0f20ba78",
  "observation": {
    "content": [
      {
        "cache_prompt": false,
        "type": "text",
        "text": ""
      }
    ],
    "is_error": true,
    "command": "C-c",
    "exit_code": -1,
    "timeout": false,
    "metadata": {
      "exit_code": -1,
      "pid": -1,
      "prefix": "[Below is the output of the previous command.]\n",
      "suffix": "\n[Your command \"C-c\" is NOT executed. The previous command is still running - You CANNOT send new commands until the previous command is completed. By setting `is_input` to `true`, you can interact with the current process: You may wait longer to see additional output by sending empty command '', send other commands to interact with the current process, send keys (\"C-c\", \"C-z\", \"C-d\") to interrupt/kill the previous command before sending your new command, or use the timeout parameter in terminal for future commands.]"
    },
    "kind": "TerminalObservation"
  },
  "action_id": "11da6fc7-560a-43fd-9df0-421ce3dced6b",
  "extended_content": [],
  "kind": "ObservationEvent"
}
It sounds like the poor thing was trying to Ctrl+C, but that doesnā€™t work
The previous one is similar
Copy code
{
  "id": "62761869-340c-4f18-996f-27a244124b70",
  "timestamp": "2026-08-01T18:33:59.564123",
  "source": "environment",
  "parent_id": "8a4531a2-6a13-438d-85a4-7fa4b9790474",
  "tool_name": "terminal",
  "tool_call_id": "2ec949ee-f416-4105-8c40-ecb0eae433f0",
  "observation": {
    "content": [
      {
        "cache_prompt": false,
        "type": "text",
        "text": ""
      }
    ],
    "is_error": true,
    "command": "C-c",
    "exit_code": -1,
    "timeout": false,
    "metadata": {
      "exit_code": -1,
      "pid": -1,
      "prefix": "[Below is the output of the previous command.]\n",
      "suffix": "\n[Your command \"C-c\" is NOT executed. The previous command is still running - You CANNOT send new commands until the previous command is completed. By setting `is_input` to `true`, you can interact with the current process: You may wait longer to see additional output by sending empty command '', send other commands to interact with the current process, send keys (\"C-c\", \"C-z\", \"C-d\") to interrupt/kill the previous command before sending your new command, or use the timeout parameter in terminal for future commands.]"
    },
    "kind": "TerminalObservation"
  },
  "action_id": "8a4531a2-6a13-438d-85a4-7fa4b9790474",
  "extended_content": [],
  "kind": "ObservationEvent"
}
This looks like a different bug: Ctrl+C doesnā€™t work?
m
I do have another example
c
Quick thought: in order to send Ctrl+C successfully, does the LLM need to set 
is_input
parameter? It didnā€™t set it
Copy code
input":false
Fascinatingā€¦ it wants to 
create
and it gives a 
path
, but not 
file_text
Quick thought: it might be interesting to run GLM e.g. a 50x instances eval, with Gemini toolsā€¦ which are easier than a ā€œfile editorā€ with sub-commands šŸ¤”
m
yes... But the think is that you can also have this kind of problem if you are running long workflow with the SDK. For that reason I wanted to solve it at SDK level
c
I agree on the STUCK issue, we should do it
Does GLM have some harness from its provider or do we know what harness was used for its own evals?
m
Does GLM have some harness from its provider
No
what harness was used for its own evals?
The GLM-4.5 technical report: arxiv.org/abs/2508.06471 From Section 4.2.3: SWE-bench Verified was evaluated using OpenHands v0.34.0 (100 iteration cap, history truncation, temperature=0.6, top_p=1.0), and Terminal-Bench was evaluated using the Terminus framework with standard function calling.
c
Interesting, thank you so much for looking it up! Oh wait, itā€™s GLM 4.5, weā€™re at ā€¦ 5.2 ?
m
I found just this one...
c
But itā€™s interesting that it was using OpenHands last year. That means it was evaluated using this same file tool
m
Ah ok there is a blog with the data about that. It was hidden šŸ™‚ - SWE-bench Pro: OpenHands (tailored instruction prompt) - Terminal-Bench 2.1 (reported score 81.0): Terminus-2 - Terminal-Bench 2.1 (best-reported-harness, 82.7): Claude Code 2.1.167 - DeepSWE: mini-swe-agent harness + official pier eval framework - ProgramBench: Claude Code Source: z.ai/blog/glm-5.2
aha 1
thanks on a stick 1
c
My thinking is that some LLMs, from Sonnet 3.6+ to some open LLMs, have been post-trained on this 
file_editor
tool. Those Iā€™ve seen evaluated on OpenHands are particularly likely to, since it was the file tool for a long time. In fact, weā€™ve even seen a Nemotron (I think) which was stubbornly calling 
str_replace
because that was the old name of this tool. šŸ˜…
In theory, they could generalize so that it doesnā€™t matterā€¦ In practice, well, you know, pesky practice messing things up
m
I start to see a lot of errors in this direction (with 
file_editor
)
c
This might be worth considering. The agent knows how to eval custom builds, it has a skill on how to eval, so I think it could make it end to end, if it makes a branch on 
benchmarks
too (I think?): https://openhands-ai.slack.com/archives/C06R25BT5B2/p1785759700460139?thread_ts=1785751907.222289&channel=C06R25BT5B2&message_ts=1785759700.460139 It's not even for the eval per se, it would be so that we see how it behaves with those tools and what kind of errors, if any, show up in a mini-eval
Not sure if I can do it anymore, since we removed the workflow dispatch and somethingā€¦ WDYT?
m
No I think you can not do this. @millions-crayon-32880 is also looking inside the problem. Perhaps it is really an inference problem
c
Just to note, before I forget: it seems temperature=1 in Zaiā€™s report, might be worth checking if we use the same
m
Hey GLM-5.2 should be fixed after we changed providers from fp4 to fp8
Ping me if it persists šŸ™‚
m
it seems to work fine
thanks šŸ™‚
2 Views
Open in Slack
PreviousNext
Powered by