https://docs.all-hands.dev/ logo
Join Slack
Powered by
First look at the Intelligent Model Router. CC <@U...
# proj-agent
m
First look at the Intelligent Model Router. CC @limited-student-10747 @average-processor-75195 It adds a tool 
classify_and_switch_llm
that classifies a task and switches to the appropriate LLM profile. The tool is configurable by setting meta profiles, which contain models pair with tasks descriptions. Still thinking the UI for suggesting a good default router. Please see screenshots. PRs: https://github.com/OpenHands/software-agent-sdk/pull/3744 https://github.com/OpenHands/agent-canvas/pull/1395
oooh 2
🔥 4
a
Great progress - for the default models you selected, is there a snippet on the performance of them (I know you talked about it easier and doesn't need to be in the PR) - looking forward to this in action
m
Assuming a perfect classifier and only assigning one model at the start of the conversation here is the score/cost graph:
Classifier isn't perfect though, and picking the wrong model can impact either cost or score.
I'll rerun the graph with the current classifier and let you know.
a
I saw GLM 5.2 on front end tasks doing well.
l
Thanks a lot! Yes, I’d also like to see the numerical result of the current classifier.
m
red line is perfect classifier, blue line is classifier v5 as quoted here
You are a task classification expert. Your job is to classify software development task instructions into exactly one of the following categories:
Categories:
1. Greenfield - Tasks that involve creating new projects, repositories, or applications from scratch. These tasks typically start with nothing and build something new.
2. Frontend - Tasks involving USER-FACING applications, web development, UI/UX, client-side code, or visual components.
Frontend includes:
- UI/Visual: CSS, HTML, styling, rendering, display, layout, responsive design, animations, visual bugs, browser rendering
- Frontend Frameworks/Libraries: React, Vue, Angular, Svelte, Next.js, Nuxt, Gatsby
- Frontend Tooling: ESLint (JavaScript linting), Prettier (code formatting), Babel, Webpack, Vite, Rollup
- JavaScript/TypeScript: Client-side JavaScript, TypeScript, JSX, TSX
- Web Technologies: DOM manipulation, browser APIs, Web Components, Canvas, WebGL
- Frontend Libraries: jQuery, Lodash, Moment.js, D3.js, Chart.js, Three.js
- Map/Visualization Libraries: OpenLayers, Leaflet, Mapbox, Google Maps API
- UI Component Libraries: Material-UI, Ant Design, Bootstrap, Tailwind
- State Management: Redux, MobX, Vuex, Zustand
- Form Libraries: Formik, React Hook Form
- Testing (Frontend): Jest, Cypress, Selenium, Playwright, Testing Library
Keywords that signal Frontend:
- Linting, formatting, prettier, eslint, syntax highlighting
- Component, element, widget, view, template
- Rendering, display, visual, appearance, layout, style
- Browser, client-side, frontend, UI, UX
- JSX, TSX, HTML, CSS, SCSS, styled-components
- OpenLayers, Leaflet, map, GeoJSON, tile, layer, canvas
3. Testing - Tasks that involve writing tests, debugging tests, fixing test failures, or improving test coverage for BACKEND code. This includes unit tests, integration tests, end-to-end tests for server-side logic.
4. Information Gathering - Tasks that involve research, data collection, answering questions about concepts or best practices, or gathering knowledge WITHOUT implementing changes. These tasks focus on finding or synthesizing information rather than making code changes.
5. Issue Resolution (other) - Tasks that involve fixing bugs in BACKEND/SERVER logic, resolving API issues, debugging data processing problems, making corrections to business logic, database issues, server-side error handling, or implementing technical decisions about backend behavior.
Guidelines:
- Read the task instruction carefully and identify the primary goal
- Library/Framework Detection: If the task mentions ESLint, Prettier, OpenLayers, Leaflet, React, Vue, Angular, or other frontend libraries → Frontend
- Frontend vs Backend: Frontend = what users see and interact with. Backend = server logic, APIs, databases, data processing
- Frontend includes: linting tools (ESLint), formatting tools (Prettier), map libraries (OpenLayers), UI frameworks
- Issue Resolution = backend bugs, API problems, server errors, database issues, business logic bugs
- Questions about "should we implement X?" are Issue Resolution if they involve backend behavior
- Questions seeking general knowledge are Information Gathering
- Testing tasks must focus on backend test creation or debugging
- When in doubt: Does this involve code that runs in the browser or affects what users see? → Frontend
Respond with ONLY the category name, exactly as written above.
Most error comes from frontend, and thus the task description is a bit too complex for my taste
l
How are you able to get a line? I thought it would just pick a single model for each instance so it’d be a single point?
m
The classifier picks a category
So on frontend, the classifier does
then I can use that over each tuple of possible models (each red dot) to calculate how it performs. Say
The 1.96% that gets routed to Info Gathering behaves the same at much higher cost. While the others perform worse at less cost.
So
So for those models Classifier frontend score = 0.849·41.2 + 0.1275·36.8 + 0.0196·41.8 + 0.0039·36.8 = 40.63% (vs ideal 41.2%) Classifier frontend cost = 0.849·0.64 + 0.1275·0.35 + 0.0196·2.37 + 0.0039·0.35 = $0.64 (vs ideal $0.64)
l
What’s the algorithm for picking which model to use for any particular instance then?
m
It's a single LLM call on claude-sonnet-4-5 that uses thie prompt above the last graph + the instance to classify.
l
But how do you go from category to LLM name?
m
Maybe I missed it, but I thought the goal was to lean on the scores from https://index.openhands.dev/ and utilize that to choose which model to use
a
If the baseline is “send every Frontend task to Claude Opus 4.6 at $2.37,” then the current classifier costs about $0.636, saving $1.734 per task, or 73.2%. Is that right?
m
@microscopic-night-41309 the red line in the graph was calculated directly from the openhands index. you can graph it by taking
Copy code
ds = load_dataset("OpenHands/openhands-index", split="test")
then selecting for each possible score, the 5 tuple that would require the least cost.
m
Oh nice! Thanks - I 100% missed it 😆
np 1
m
@average-processor-75195 baseline cannot be against frontend only, but it can be "send every task, no matter its type, to Claude Opus 4.6" That baseline has a score of 66.7% and costs $2.37. You can get same score of 66.7% at $1.14 Saving $1.23
🤯 2
@limited-student-10747 that depends on what score you are aiming at, but we can define different meta profiles that use the same classifier but route at different models.
maybe we could have two defaults. One that uses Opus-4-6 as a baseline and another that uses Sonnet-4-6 as a baseline.
Same classifier but different models to route to
l
I mean specifically how did you calculate the accuracy here? In order to use the agent you need to select an LM, right? Then you run the agent, get the result, then grade it. I'm wondering how you did the "select an LM" part when you created this graph
m
Ah! got it! I calculated the accuracy of the classifier by running it against all instances to classify them, but I did not evaluate the instances once classified. So I'm assuming two things: 1. the classifier behaves the same when run in batch as it would when run as a tool in the first step. 2. the instance will get the same score / cost as the model it was assigned to
l
Yeah I think we don't need to re evaluate the instances, but I still don't understand how you select an LM to use after the router selects a category
m
I'm not sure I understand the question, but in agent-canvas we would save a meta-profile like
and the 
classify_and_switch_llm
tool will classify and switch when asked to
l
Ok I can write up a description tomorrow!
m
Happy to help!
e
This is so cool Juan!!
ty 1
The task classes idea is a great idea for helping customize the experience
Quick question, how are we thinking about this for OHE Orgs
‼️ 1
m
@elegant-branch-54423 current proposal creates a meta-profile, similar to existing LLM profiles. Ideally OHE would treat them same as regular LLM profiles.
e
Interesting. I think @curved-angle-80449 or @dry-car-37037 was interested in this!
🙌 1
d
really cool stuff!
until we merge the agent canvas as a OHE frontend, integrating this will likely involve porting over the code to the current saas ui and making sure it works in different scenarios like org/instance allowed BYOK, admin-only llms options, etc. also in general the llm profile code is super buggy because it doesn't have a real backend state machine so eventually we should add that to support llm profiles and this new llm router
probably would need a good look at for OHE but overall very exciting and an extremely clear example for users and enterprises of our unique value props
gonna be exciting when we can use this feature to generate and show off side by side llm cost (and task performance) numbers of OHE with this feature vs something closed source like claude code or codex
nod 1
m
It should be straight forward to support in current SaaS again, but notice that it is not merged in agent-canvas nor sdk yet. we are currently on proposal phase with Graham.
d
would this likely need new in-app conversation frontend changes as well? so users can see what's happening in their conversation when this is router running?
m
for sure, at the moment it uses existing patterns for indicating tools + indicating swapping model manually
d
nice
2 Views
Open in Slack
PreviousNext
Powered by