Another #TODO for a few tests.

# TODO: We should add test cases for all these scenarios

github.com/OpenHands/software-agent-sdk/issues/4538 github.com/OpenHands/software-agent-sdk/pull/4536

just a comment. We do have a class TestLLM that you can use to create mock test. I don't know if this apply here.

Out of curiosity, what LLM(s) do you use for these investigations, and for the PRs is it the same?

TLDR: Not sure. It will forever remain an inscrutable mystery. 😂 I say this a lot but I'm very ADHD. I should try another medicine, the last one came with crippling nausea and I stopped. I've done way more testing and juggling than sane people would do. I've read you kinda need to have the gift of hyperfixation for these things. Anyway, I used to do shootout competitions between models. My opinion is that most users do not realize how much impact being probabalistic has in reality. And just imagine all the little MoE gate conditions or whatever. I found that the model that was an absolute genius on one task may be unable to compose a hotdog recipe two hours later. Or more importantly, it may fail gloriously at that same task. This morning's work could honestly have come from sessions with Claude, Codex, or Kimi. However, if any serious code was probably sent to M3 and then reviewed by GLM-5.3 using my in-house "delegate" command. I have a tool that built for vetting anything. It was built working on OSS projects. Don't want to bore you to death and I may have talked about it before. All the SOTA models in one harness? It's overkill but offers a mountain of perspective.

On a side note, the reason I ask and personally, care about this, even idly contemplate asking for LLM and Agent by default in PRs (for example, idk, maybe we could ask this question in the PR template), is that the prompt and LLM matter for various issues. For example, if the agent makes some mistakes, I like to know what LLM and with what context, to ~know how to fix them; or before that, to have some intuition of where the risks are. Then again, I’m a bit of an LLM freak, to put it… mildly. 😅 TBH I also think - and see, out there - that we could somehow save the prompts and LLM/agent details, as documentation for the future. I don’t necessarily mean full histories; that’s a lot and has its uses, but what I have in mind is simpler… the prompt, the agent and LLM, a descriptive summary of the agent’s turn. It would enable easier review I feel; assessment; continuation; control. Sorry for brainstorming on the fly! And thank you for the tests, we merged the PR now 🙏

I'm happy to tell you every detail about my setup. Idc; we can jump on a call and go through it. Heck, I'd get all of my internal tooling cleaned up for visitors and open source it but I doubt much is novel. There are probably commercial alternatives for all my bespoke solutions. The model filling out the form will be happy to announce itself, I'd imagine. In my case, the provenance would be untraceable, tho. But I'm definitely not the typical user. If K3 is filling out the form, she'll be happy to take credit for everything including the creation of the universe and that would obscure a more nuanced history.