Quick question, what’s the easiest way to evaluate...
# proj-agentl
limited-student-10747
12/19/2025, 8:00 PMQuick question, what’s the easiest way to evaluate a new agent variant now, can we do it in CI?
I’d like to evaluate whether swapping out our standard file editing tools with gemini file editing tools results in an increase of performance when using Gemini.
https://github.com/OpenHands/software-agent-sdk/pull/1199/changes
c
chilly-france-44439
12/19/2025, 8:04 PMtag:
intergration-testbehavior-testchilly-france-44439
12/19/2025, 8:04 PMbut we don't yet have tests where gemini will fail, so i guess this will just verify that it works, not that it will increase performance
l
limited-student-10747
12/19/2025, 8:07 PMyeah, I’d actually like to run an eval, and also it needs to use alternative tools: https://github.com/OpenHands/software-agent-sdk/pull/1199/changes#diff-b58ffa5ee7c9401a387760001743d289cb1c4988ae64e1a874c76510cd411c73R33
Maybe the only way to do that is run locally?
limited-student-10747
12/19/2025, 8:07 PM(I already ran the integration test with these tools and they all passed)
c
calm-artist-7206
12/19/2025, 10:11 PMMay I suggest that in this case, just like GPT-5, we could merge the tools if the implementation is ok and they work, and just leave them as non-default until proven to increase performance?
It will mean that people can use them, and maybe even we could make a preset, just not the default preset
calm-artist-7206
12/19/2025, 10:12 PMand then maybe they can tell us if they work or can be improved
l
limited-student-10747
12/19/2025, 10:15 PMOpenHands is running an eval for me now :)
acknowledged 1
c
calm-artist-7206
12/19/2025, 10:16 PMLooking forward to the result!
2 Views