Fun experiment with one of the OpenHands trajectories in NVIDIA's Open-SWE-Traces dataset: instead of asking "did it pass?" (SWE-bench already answers that), I asked
how much of the run an independent reviewer could actually reconstruct from the trace alone.
Answer: ~29%. The other 71% isn't the agent's fault — it's events that simply don't exist in the trajectory (no wall clock, no actor roles, no approval/acceptance events, no resource refs). So the profile turns each unanswerable question into the specific event that would make it answerable — basically a free instrumentation wishlist for agent frameworks.
Both profiles public + reproducible from the dataset:
factnotebook.com/profiles
Curious if the events it asks for are things OpenHands could plausibly emit — or which ones you'd call wrong-headed.