https://docs.all-hands.dev/ logo
Join Slack
Powered by
Is there a problem with running automations on for...
# proj-automation
c
Is there a problem with running automations on forks? Stuck Cloud Reviewer This never finished, twice apparently?
m
I can give a look
conversation_a62354d1-7e0c-4665-9095-89e7bd66deb8.zip,conversation_a62354d1-7e0c-4665-9095-89e7bd66deb8(1).zip,conversation_c1b6c3d9-d116-49fd-9891-3fb2b5894b98.zip
conversation_c1b6c3d9-d116-49fd-9891-3fb2b5894b98.zipconversation_a62354d1-7e0c-4665-9095-89e7bd66deb8.zipconversation_a62354d1-7e0c-4665-9095-89e7bd66deb8(1).zip
But it is difficult to say what happened... both produced no further events... and stopped mid turn
nod 1
c
It seems one is Minimax-M3, and the other is GPT-5.5 !
That's weird.
Oh, 
smolpaws
is telling me that Minimax “never invoked the code-review skill”, while “gpt-5.5 followed the code-review skill correctly”
Ah, these conversations don’t have the system prompt 🤔 GPT-5.5 used invoke explicitly anyway
b
🐾 smolpaws here — I dug through both event-log exports (the 🐱 that Engel mentioned). Here's the fuller story, because my first take needed correcting. Same trigger, same entrypoint for both runs. 
pull_request.labeled
, label 
review-this
, PR #3557. Both were handed the same 
AUTOMATION_EVENT_PAYLOAD
and the same prompt (Minimax's own reasoning at one step literally says "I have the payload from the user prompt"). So this is a clean same-task, two-model comparison: GPT-5.5 (
c1b6c3d9
) vs Minimax-M3 (
a62354d1
). Both hit the exact same environment glitch at the start. 
$AUTOMATION_EVENT_PAYLOAD
came through empty/garbled, so the first 
python3 … json.load(...)
threw a JSON traceback in both runs. Identical failure on turn one. What each did next is the whole difference: • GPT-5.5 invoked the 
code-review
skill on turn one (before even touching the payload), pulled in the full ~15K-char reviewer persona + grounding rules, and then started working the diff. Skill-first — so it had the method loaded before the mess. • Minimax-M3 went straight at parsing the payload, hit the traceback, and then spent ~25 turns firefighting the environment: wrong token var name (
github_token
vs 
GITHUB_TOKEN
→ a 401), "shell rejecting multi-line python," a script that concatenated both file diffs into one, then re-reading files to recover. It never reached the "how should I review this" step, so it never invoked the skill. Corrected conclusion (my first message overstated this): Minimax issued no code-review skill tool-call — that part is confirmed, 0 fingerprints anywhere in its 117 events. But it didn't "ignore" the skill out of preference; it was almost certainly offered the same skill and simply got lost in setup and never got there. GPT-5.5 sidestepped the whole mess by reaching for the skill first. Why neither finished: both ended mid-turn with no review posted, but for opposite reasons — • GPT-5.5 ran out of room/cost doing the review properly: per-turn context climbed to ~83K tokens, ~$1.12 in 6 actions, condenser not compressing. • Minimax-M3 burned its turns just trying to orient — cheap (~$0.07) but never escaped the plumbing. The actionable bit: the empty 
AUTOMATION_EVENT_PAYLOAD
+ the 
github_token
vs 
GITHUB_TOKEN
mismatch looks like a real harness bug — it tripped both models on turn one, and the weaker one never recovered. Worth fixing the payload injection + standardizing the token env var name; that alone would've saved Minimax ~20 wasted turns. (Caveat: these exports contain no system/user MessageEvent, so I'm inferring the shared prompt from the agents' own reasoning + the identical trigger, not reading it directly.)
🐾 Correction to my own post above — Engel rightly called out my "why neither finished" section. I was wrong there: • The ~83K-token / ~$1.12 framing for GPT-5.5 is not a stop reason — 83K is nothing these days, and there's no budget/context event. Scratch that explanation. • "Minimax ran out of turns orienting" was also speculation — that describes its style, not why it stopped. What the events actually show about the stop: in both runs the final event is a tool ObservationEvent with 
exit_code 0
(a successful command), followed by no agent action — and crucially no 
FINISHED
event. A normal run ends with a FINISHED state. Neither has one. So both were cut off after a clean tool result, before the next LLM turn produced anything — i.e. abnormal termination, not the agent choosing to stop and not a cost/context limit. The likely culprit is whatever happens between that last successful observation and the next model response (the run dying / the post-observation LLM call never landing), which fits the "stuck, never finished, twice" symptom better than anything model-side. tl;dr: the model-behaviour contrast earlier (GPT-5.5 skill-first vs Minimax stuck in payload plumbing) still stands from the events. My guesses at the termination cause don't — both just stop dead after a successful observation with no FINISHED event.
m
It seems one is Minimax-M3, and the other is GPT-5.5 !
This is normal. Everytime another model is assigned
c
Why? Among what models?
m
Becuase AB-testing cc @wide-doctor-37830 @quick-window-91705
c
I started another thread on this here https://openhands-ai.slack.com/archives/C0B4Z32L064/p1782204421426049 Sorry, that’s a different issue. I would like to know if I’m wrong
Open in Slack
PreviousNext
Powered by