https://docs.all-hands.dev/ logo
Join Slack
Powered by
I could use a review on the sdk release: <https://...
# proj-agent
l
I could use a review on the sdk release: https://github.com/OpenHands/software-agent-sdk/pull/3374
👀 1
✅ 1
c
It still looks - sorry, typing on the run - like something broke Gemini, and apparently we still have a budget error for Kimi. (1) Maybe we can drop support for Kimi if so? Or do we need it? (2) For Gemini though, could we investigate what broke it?
Sorry, I don't think I can help for ~ a few hours. Just to note, every time we made releases before, e.g. up to recent weeks I was here, and we fixed LLM errors until the release tests passed because they indeed pass. I think it's a good practice to have on the repo, and I'd suggest to not relax that condition, it's important IMHO. I can try to get to it asap
l
oh sorry, honestly i didn’t notice the gemini issues. we can try to fix that
c
On a side note, we may want to do something about the agent's comment spam - the PR gets very hard to read. I think maybe this should help a little: (it removes the duplicate comment) https://openhands-ai.slack.com/archives/C07V9BMGHH6/p1779477187766909
Re: Gemini, lets make a separate PR, it sounds like it needs a fix and I'm confused why eval worked, too
l
Ah, that is indeed the proxy version we’re running for evals I think.
c
We need to fix this release process a bit. I'm preoccupied because I reported Gemini in the previous release Slack thread, I requested changes on the PR on Gemini, Kimi, workflow, and also, the integration tests kept failing all of Gemini (it's in the integration tests comment), and yet: your agent said like
Re-ran integration tests. The label-triggered default run exposed provider/model failures (Gemini payload rejection and Kimi provider balance). I then re-ran the same stable model set used by the prior successful release check: ✅ 100% on DeepSeek V4 Flash, GPT-5.5, and MiniMax M2.5
And it was happy. Pretty green checkbox. It discovered how to get 100% : by running DIFFERENT models. 😅 It's a Claude? How can we do this better?
l
Actually my agent is minimax-m2.7 now, so not super-surprising it’s making stuff up
🫠 1
yeah, I think we need to fix the release process. we can probably remove Kimi
(otoh, i’m reasonably impressed by minimax 🙂)
c
On a side note, I think maybe this also shows we don't have enough guardrails to do it automatically. We still needed to read the comments, and we need to make the choices. And I think that's OK. It's a release; of course the humans take responsibility for it, as much as I like my agent, I don't think it's ready lolsob Though on my personal github, I let the agent make releases 100%! But that's a playground, not a project used in production 🙏
l
Yeah, I agree!
c
we can probably remove Kimi
• PR replacing Kimi with Minimax M2.7
🙏 1
Is it possible to fix the eval proxy, just a version up is enough?
to litellm 1.84.1
Looks like my agent got it to work on app, so I think it's low risk considering the previous release patch was only days ago
l
Yeah, I think we should increase the litellm eval proxy version. @nutritious-balloon-65098 open a PR on All-Hands-AI/infra increasing the eval proxy version to 1.84.1
n
I'm on it! Graham Neubig can track my progress here.
Since my last summary, no additional changes were made. Checklist: • ✅ The requested eval LiteLLM proxy version update was completed. • ✅ Changes were committed and pushed to a new branch. • ✅ A draft PR was opened: https://github.com/All-Hands-AI/infra/pull/1271 • ✅ The PR includes an AI-agent disclosure. • ✅ Validation was run before opening the PR. • ✅ No extra or unrelated files were changed. The changes are concise: a single-line image tag update in 
k8s/evaluation/litellm.yaml
.
c
I made a few more checks and I think we're good
Maybe we can tell the reviewer something in stronger language about release PRs... idk, it shouldn't let it pass when any of the 3 release test labels fail
l
Thanks so much for the close look!
5 Views
Open in Slack
PreviousNext
Powered by