Hey 👋 I did some brief research into the bugs we've been seeing in the SDK and why we keep introducing them (see attachment). Having bugs is normal given our complex codebase and the challenges of working with APIs. I wanted to understand what we can do to prevent them, or at least reduce their frequency. There are a couple of actionable takeaways we can address quickly. To get started, I’ve opened a few small PRs. (others are more complex and big changes). I’d like to revisit this in a couple of weeks to see if we've managed to reduce the incoming bug rate. 🙂 To review: • ci: run downstream test suites when upstream packages change • test(cross): check LocalConversation and RemoteConversation parity • fix(sdk): align RemoteConversation errors with BaseConversation • test(sdk): drive pause tests with TestLLM instead of patching litellm • docs(agents): require a failure-path test for each changed behavior

The list of unrepresented test types is very interesting. I was looking at some of those, I think we can set them up nicely. I don't agree with the agent that more unit tests = good... It's an interesting one. Of course that's going to be on top, since a test could catch anything. But agents modify tests. They modify the old tests, many many times, tests are not slow to modify as they used to be in the Before Times... I should write more about this

I don't agree with the agent that more unit tests = good...

I do think it is the case. But we should not give the agent possibility to write its own tests. The two task should be independents and in a loop. So you create the feature and then another the tests and then you iterate

I think the worse part is, LLMs write unit tests coupled to the implementation at that point in time... Next time some detail needs to change, they change the tests. If so, I see two options: either we go back to a human reviewing unit tests, or we trust another LLM to... say that the change was okay and go towards automerging the new bunch of test-only PRs that we'd get?

yes this is a problem... but it is ok if it changes the tests in a good way. My point is that there are a lot of PRs that introduce bugs and people don't check a lot. So we should create an env where we are sure that not too many bugs are introduced. It is impossible to not introduce bugs but what can we do to prevent most of them?

✅

One thing I've asked the agent recently, was how many times did a PR add new tests and, in the same PR, change old tests. It was some unexpected number,

• 300+ where new tests were added, in the same with old tests changed

• 350+ where new tests were added, without changing old tests

@calm-artist-7206 This is a very good point 🙂 Actually I did a deep search in internet and this is not measure, i..e, are LLM changing tests such that the PR pass the CI? (and so it is merged?) I'm running this analysis on our PRs Answer of the agent about that:

"Prior work measures test tampering in benchmarks or training rollouts, and measures test modification and assertion removal in real PRs syntactically. To our knowledge, none links test-weakening edits to CI state in real PRs."

Here's my thinking… Everything is changing, and a lot has already changed, since AI agents jumped several milestones since last year. I do think we need to question everything, to rethink everything. Assumptions, good practices, … not all are quite right, anymore; in fact, I think some flip. Please consider the following just brainstorming. The testing pyramid Some practices, that have become a staple of SW development: “The testing pyramid” , the idea that generally, it’s normal to have • many, as many unit tests (base of the pyramid) • some middle number of service tests • fewer integration/end to end tests. (top of the pyramid) Since, well, writing unit tests was faster and they’re small enough typically, they can become many in a codebase; on the other hand, e2e is quite involving, long and time consuming, both as writing and as running were concerned, back in the day. Boring to wait for them on every local development run - so they were “exceptional”. That’s… not true anymore. We can have the agent run everything end to end, and prove it all, while unit tests… something funny is happening to unit tests: LLMs modify them... a lot, a lot of times. Let me look at a few other SWE assumptions: “Unit tests prevent regressions.” “Unit test coverage is good, more coverage is good.” “Green CI feels safe.” I simplify a bit (we can have some caveats), but mainly I do think we believe something like this. We believe it, and we were right to believe it in the past. It used to be fairly accurate… when we were adding code at the speed of a human writing it. “More tests are good, because they help preventing regressions” OK. What is a regression? Unexpected behavior, of a part that used to behave as expected. OK. So it’s about human intention. The Old Times This was definitely true in 2024. Let’s say I was changing behavior of the

stuck

check, and some old unit tests failed. If 5 failed, I was looking at each, one by one, in my editor, and decide: oh, those 2 actually should fail, but those 3, uh, shouldn’t have, it's a regression. So I fixed the 2 tests, and fixed the implementation for the other 3 to pass. I chose the new behavior: the one where 2 old tests needed modified, too. At the speed of a human reading and editing. At slow human speed. Slow is what allowed us humans to rely on tests: that I was looking into every red and chose what will make it green. To note, I had the agent, OpenHands agent, doing stuff, but it was so amusingly hard to get it to run all tests, and fix all tests, nah, it kept declaring success with a few! 😅 Fast-forward to today… We are not reading unit tests. I started to notice I’m skimming them in September 2025. And that is many LLM eras ago. Humans are not going to read code, let alone unit tests. In the same time, agents became so good. They always get tests green now. They always get them green. The human doesn’t stop anymore, doesn’t painstakingly look at each of those 5 unexpected fails next time, let alone decide every bit of behavior. The agent chooses what to do with those 5 old tests. The agent chooses the shape of the new behavior. Did it choose what I intended, the human intention, or not? I find this question fascinating. I… guess we trust them? I think it means we trust them, in a way, to know and choose what the human intention must have been. Because we’re obviously merging, and we’re not really reading tests, and we feel good when CI is green. It feels safe when CI is green. 😅 Thoughts going forward… Code has become more fluid. Tests’ code has become more fluid. We still call them deterministic, but… they’re fluid. I don't know if agents are weakening them, or change them in ways humans don’t, actually, intend. GPTs, at least until 5.x, treat many old tests as sacrosanct, and they’d put a bunch of hacks in the implementation code, just so they pass without modification. That ain’t great either.🙊 I do think these beautiful models are pretty good, though; as they take the old tests in context, and they’re heavily post-trained on tests, they can make ~reasonable choices on what the new behavior is. 🤔 I think of tests as prompts. Context snippets, in a formal language, nudging the LLM in a direction. Not as… “deterministic tests”… I mean, hey, they’re fluid. But I still want to know and verify the behavior. What I like a lot, is semi-strict context, like invariants in some long-forgotten language, scripts with agent-friendly CLI commands (e.g. verify API specs ), rather than fluid unit tests. (with some exceptions, e.g. unit tests for a parser continue to be great) I feel the Testing Pyramid has inverted. We can now run everything live, like we didn’t think possible before. End to end tests, real end to end, no mocks, where the agent uses the application / API as the end user does, are now on the base of the pyramid - we can set them up and run, I do think we can, actually, run all behaviors. So, it flipped… It’s like: • base of the pyramid: e.g. verify-agent-server / daily self-healing, 800 + 950 + n user-facing behaviors • middle: property tests / experiment with some “niche” types • top of the pyramid: good unit tests: well targeted, etc - by the book.

┌─────────────────────────────────────────────────────────┬─────────────┬────────────┐
│                          Area                           │ Tests added │ Bugs found │
├─────────────────────────────────────────────────────────┼─────────────┼────────────┤
│ llm / tool / mcp / security                             │ 34          │ 7          │
├─────────────────────────────────────────────────────────┼─────────────┼────────────┤
│ agent / conversation / event / critic / context / hooks │ ~71         │ 8          │
├─────────────────────────────────────────────────────────┼─────────────┼────────────┤
│ workspace / cross                                       │ 56          │ 10         │
├─────────────────────────────────────────────────────────┼─────────────┼────────────┤
│ misc sdk (io, git, skills, plugin, utils, …)            │ 79          │ 26         │
├─────────────────────────────────────────────────────────┼─────────────┼────────────┤
│ agent_server                                            │ 32          │ 6          │
└─────────────────────────────────────────────────────────┴─────────────┴────────────┘

I just asked a flew of agents to add unit tests for non-happy paths

For the record, on the OpenHands repo, my agents continue to find bugs, regressions or new bugs introduced by new PRs. And they submit PRs the next day.

haha yes I know

Oh. We could slow down PRs and run the related behaviors per PR, if that’s important?

@calm-artist-7206 github.com/OpenHands/software-agent-sdk/issues/6348 Do you want to use your storm of agents to fix them? 🙂