Fun experiment with one of the OpenHands trajector...
# random
a
Fun experiment with one of the OpenHands trajectories in NVIDIA's Open-SWE-Traces dataset: instead of asking "did it pass?" (SWE-bench already answers that), I asked how much of the run an independent reviewer could actually reconstruct from the trace alone. Answer: ~29%. The other 71% isn't the agent's fault — it's events that simply don't exist in the trajectory (no wall clock, no actor roles, no approval/acceptance events, no resource refs). So the profile turns each unanswerable question into the specific event that would make it answerable — basically a free instrumentation wishlist for agent frameworks. Both profiles public + reproducible from the dataset: factnotebook.com/profiles Curious if the events it asks for are things OpenHands could plausibly emit — or which ones you'd call wrong-headed.
🙌 1
👍 1