Hey everyone - I’ve been working on a fault-inject...
# general
s
Hey everyone - I’ve been working on a fault-injection reliability benchmark for OpenHands agent runs: RFC: github.com/OpenHands/benchmarks/issues/764 Implementation PR: github.com/OpenHands/benchmarks/pull/765 The benchmark runs matched baseline/faulted tasks under deterministic sandbox restarts, lost dispatch responses, SIGKILL mid-tool-call, and network partitions. It produces inspectable evidence for: • whether the run resumes and completes • whether replay duplicates an irreversible effect • how much recovery overhead it adds The benchmark core, graders, SDK event bridge, CLI, docs, and tests are implemented. Runtime-specific fault operations remain behind explicit adapters. I’m looking for guidance on two decisions: 1. Is `OpenHands/benchmarks`the right long-term home, or should part of this live in the SDK? 2. Which real runtime/SWE-bench adapter would be the most useful first integration? If anyone owns benchmark infrastructure or runtime recovery, I’d really appreciate a pointer or review. I’m happy to adjust the scope around what would be easiest to land.