Hi everyone.. I've prepared a snapshot_replay agent which can execute agent actions from a persistent conversation without triggering any actual LLM calls. Developed it as part of regression test pipeline. Hope this would be useful to someone.. Please find the implementation here

could you open a draft PR so it will be easier to see the diff? 🙂

Ah, I see. Actually, we already have something similar in the SDK. If you look at

openhands-sdk/openhands/sdk/testing/test_llm.py

, you’ll find the

TestLLM

class. This is the only primitive you need to mock responses, as only the base

LLM

class makes actual API calls. Since

TestLLM

extends the

LLM

class, any changes made to the base class are automatically reflected there. I get the impression that the gap the 'replay agent' is trying to fill is: 'I have a long, real recorded trajectory and I want to use it as the oracle without hand-writing assertions.' The only thing

TestLLM

doesn't provide out of the box is a recorded oracle for observation comparison. However, that feels more like a fixture/utility concern than a reason for a new

Agent

subclass. We could achieve the same outcome with a ~50-line test helper: load the events, feed each

ActionEvent

via

TestLLM

, and diff the resulting

ObservationEvent

against the recorded one. PS: We don't use this class so much, so I can see how it might have flown under the radar.

Interesting.. I agree with your observations. I did a surface level exploration on the

TestLLM

class. Looks like a better choice than dedicated Agent subclass. In fact, it is a much better solution in terms of scalability for test cases. I will try to migrate my regression suite to this class. Thanks for the information @modern-fireman-99755