cc @limited-student-10747 @hundreds-twilight-32833 This is the combination with

pstack

, in fact mostly based on pstack for the flow

I'm still going through the 60+ bugs it found (or it thinks it found). The interesting thing is that if it thinks wrong, it’s the description of features most likely, which needs tweaking

Re: 1. I wondered the same. I think it’s likely helping the agent right now, because it’s so much to do and to cover that maybe giving it shortcuts (which is what this CLI is) is worth it to keep it on track… I'm not sure though. We could make it do it all using browser…

Re: 2. Here’s my thinking, sorry it’s a bit unbaked. In 6+ months, manually managing tests is not going to happen. I don’t see a future where humans will read the tests in CI and one by one say yeah, this is correct, no, the other isn’t. It’s not even the present, I would say.🤔 It's not even the present for unit tests, and while it may seem like the present for e2e tests, I’m not sure that’s what we do. I think the agent is making them now too. Personally, I don’t think I know off hand what e2e tests test; I know we have some, I don’t know what they do - and specially, I don’t know what they miss doing. The agent made them, the agent modifies them if they break, they become green, and I’m not sure humans always still read why. So why not give the agent the right instructions instead, the intended behavior definitions, and let it make and run the right tests on the fly?

Side note. The beauty of it, and what it’s messing my poor brain… is that this changed wildly. In ~March 2025, what I just said would have seemed crazy. Back then, Sonnet 3.7 was rushing to declare “Done! ✅”, when it wasn’t done. And it was creating tests with assert True is True // placeholder to pass the test. 😅

That's many lifetimes ago in LLM capabilities. It’s just not a thing anymore. Today, we’re in a vastly different world. I'm just reusing something we already know. We do know the agent can run OpenHands. Of course. It was running it for long, we just kept it confined to particular conversations. But it can do much more. If I ask the agent today to test a feature, and tell me in English what it did, it will tell me. It will write it in a morning report. It will tell the truth. Maybe 99.99% if we want to account for funny HF-style surprises. Comparatively, the e2e coverage is far far lower than 99.99%. So why would we prefer something with lower coverage? We can just run the entire app. In a sense, what I’m heading towards, is adaptable e2e. It accounts for what features will change this week, write up the new behavior, run the new behavior, submit issues if any, tell it in English, rinse and repeat.

If we ask the agent to run something denovo every time, its coverage of the functionality seems like it’ll be pretty inconsistent

I think the skill’s feature map is really large, and the agent drove through it fairly well. We’re not just telling the agent “test this app”, my intention here is to have it fairly detailed. I do hear your doubt, though, I have it too! I think maybe it’s okay if we take some time to see what else we need and if we can develop trust that it really does it 😅 Meanwhile, nothing is removed, we can also continue as usual 🙏

Of course, it also depends on LLM, too. I see a case just now, again, where the default reviewer running Deepseek 4.1 Flash approved the PR, while an additional review from codex with GPT-6.1-Sol uncovered four issues. Idk, as example how we could improve this, we could also run a review of sorts on the skill’s run artefacts, to see if it did it right, or something was skipped/disputable.

@limited-student-10747 I had the agent describe the CLI here: control-openhands : a CLI for agents driving OpenHands I verified most of what it claims, it seems correct to me. It has a comparison section with existing e2e: • basically the e2e tests assert about 10% of the behaviors in the feature map (68 of 715) • and e2e checks some API-level details that the map leaves out