Hey everyone - I’ve been working on a fault-injection reliability benchmark for OpenHands agent runs:
RFC:
github.com/OpenHands/benchmarks/issues/764
Implementation PR:
github.com/OpenHands/benchmarks/pull/765
The benchmark runs matched baseline/faulted tasks under deterministic sandbox restarts, lost dispatch responses, SIGKILL mid-tool-call, and network partitions. It produces inspectable evidence for:
• whether the run resumes and completes
• whether replay duplicates an irreversible effect
• how much recovery overhead it adds
The benchmark core, graders, SDK event bridge, CLI, docs, and tests are implemented. Runtime-specific fault operations remain behind explicit adapters.
I’m looking for guidance on two decisions:
1. Is `OpenHands/benchmarks`the right long-term home, or should part of this live in the SDK?
2. Which real runtime/SWE-bench adapter would be the most useful first integration?
If anyone owns benchmark infrastructure or runtime recovery, I’d really appreciate a pointer or review. I’m happy to adjust the scope around what would be easiest to land.