Can I get 👀 on 3 PRs for workspace snapshotting for evals?
Tracking issue
github.com/All-Hands-AI/infra/issues/1444
1. agent-server archive endpoint
github.com/OpenHands/software-agent-sdk/pull/3867
â—¦ added both full tar-gz and git-delta = we can refine later once we have prod data
â—¦ tar-gz has filters so we dont store too much data = we can refine later once we have prod data
â—¦ validated on real evaluation-proj benchmarks data
2. runtime-api — primary capture path
github.com/OpenHands/runtime-api/pull/623
◦ Limitations: only fires on the idle→pause/stop path, so it misses explicitly-deleted running sandboxes (→ PR #3, below)
3. app-server — secondary path + initial state
github.com/OpenHands/OpenHands/pull/14953
â—¦ archives before an explicit delete of a running sandbox (closes the gap PR #2 leaves)
◦ captures the initial state — a tar.gz snapshot taken right after clone, before the agent acts.
- Design choices: initial snapshot is a self-contained tar.gz (so it doesn't depend on base_commit re-resolving later); independent toggle RUNTIME_FILE_ARCHIVE_INITIAL_ENABLED; only fires for repo-backed conversations.
Limitations
•
initial-state placement (before vs after .openhands/setup.sh) is an open call — currently before (repo as cloned, no installed deps).
• capture is one snapshot per sandbox, keyed to the conversation that created it. Under sandbox grouping, several conversations share one PVC, so you get a single cumulative snapshot tagged to the creating conversation and the siblings get empty deltas — an attribution floor in the orchestration layer, tracked in infra#1444 (accept-vs-extend decision).
@limited-student-10747