@millions-crayon-32880 implementing benchmarks retry by 1. copying all artifacts from gcs 1 to new gcs folder 2. rerunninng infer on selected instance ids Because we dont control the eval harness, right now retry is only for infer stage. Eval will always reevaluate everything. We could do retry-eval in a future PR by having eval artifacts merge logics per benchmark (or just for sw*bench where eval is actually painful) https://github.com/OpenHands/evaluation/pull/490

@strong-ice-84145

right now retry is only for infer stage

but PR seems to have retry scope infer, eval and infer+eval?

sorry what I meant is: when you retry infer, you skip all the correct infers. But when you retry eval, you must redo all eval.