github.com/OpenHands/software-agent-sdk/pull/6068 feat(llm): add DecisionModel with OTEL tracing

I don't know if we should do that. I mean, why decision model and not ranking models? Or vision ones? or other? I would like for the sdk to stay in scope. This add just complexity

We have specific customer ask around this and we are going to need decision models for automations. I already have concrete use cases internally and from customer.

ok but you don't need to have that in the sdk

Decision models are consistent with SDK being about coding agents.

What have to do decision models wit coding agents?

There a lots of use cases for decision models in coding agents. One that I encountered personally is that when you are evaluating what a coding agent should do next on a bunch of PRs that it is managing you can pass content into the decision model and get next steps much more quickly, cost effectively and repeatedly than with an LLM. A customer of ours building on top of the SDK also found this and related coding agent use cases in their software factory. They complained that because we don't have decision model support in the SDK the OTEL traces for the agent were missing the decisions. LiteLLM already supports decision models. Another place we can use use decision models is more effective skills activation. There is quite a bit being done in this space.

A customer of ours building on top of the SDK also found this and related coding agent use cases in their software factory. They complained that because we don't have decision model support the OTEL traces for the agent were missing the decisions.

Why they are not calling the API directly? If you are using the SDK, then you have python and you can use litellm out of the box

Another place we can use use decision models is more effective skills activation.

Do you have benchmarks for that? why is more effective? Actually I benchmarked that on MetaTool and it seems that it is more or less equal to have an LLM-as-judge

Our customer shared some of their results with me and they are getting some steps done in their coding agent at 1/100th the cost and with sub second responds for steps that used to take minutes. They can and did write their own Python code, but then it doesn't have the OTEL support we offer and they have to copy it. Just doesn't make sense that we wouldn't offer this to me. Agent Zero does OpenClaw does LangGraph Deep Agents offers now Microsoft Agent Framework has added it

also I'm not against. It is just I want to stress the idea before merging because coming back it is difficult then 🙂

100%

I was thinking that we should host one of these models

If we have reservations now is the time : )

liquid ai is doing really great models

Thanks for jumping in to look at it. Since I knew you had been playing with these models I thought of you when I was thinking who to ping for reviewers.

I'm reviewing

Thanks for the thorough review! All requested change are in.

Hey, I added some tests into your PR hope you don't mind. I'm a little bit on bugs this last time and as you saw mostly bugs comes because we don't have an unit tests covering that part. And actually it catches 2 bugs haha 🙂 I didn't want to block you as I don't know if these bugs are just corner cases. So I added to xfail tests to not forget about that. I accepted the PR

Thanks for the new tests! Since you are already looking at this stuff do you want to look through the docs PR or shall I ask someone else for that? github.com/OpenHands/software-agent-sdk/pull/6339/changes

give me a moment

Yes. I was thinking we could merge the docs now with my testing of JEV and follow up with the automated running of these examples in another PR. If we want CI to run those examples too out of the gate, we'd need to solve for that. I think we already have OpenRouter API keys used in one or more repo, no? JEV is available from OpenRouter and we could wire it up that way.

We are running these tests on every release. So this will make the test failing.