Hey team! I want to introduce you to a tool that I use with OpenHands. Its called Maestro and it intelligently delegates tasks to different models and harnesses depending on complexity and size of the task. The architecture is something like this:

Maestro

            Task arrives
                 │
                 ▼
          Analyze complexity
                 │
      ┌──────────┼──────────┐
      ▼          ▼          ▼
 Claude Code  OpenHands   Cursor Agent
      │          │          │
      └──────────┼──────────┘
                 ▼
            Validation
                 ▼
        Architecture review
                 ▼
         Benchmark result
                 ▼
       Learn for next time

Do you have any tools you use with OpenHands that you recommend? I'm always looking for new/better ways of working 😄

Awesome! @millions-crayon-32880 is working on model routing and has some good experiments on it, we’d love to get your feedback when we release it soon

Very cool! I look forward to trying it. It's definitely where we are at as an industry, it only makes sense that we would respond to token maxing with model routing 😄

Just for awareness, some didn’t manage to make it work: 😅 https://manifest.build/blog/why-we-deprecated-our-llm-router/

Thanks for sharing this blog Engel - it is a nice read! Given that the API costs after prefix caching are upto 10x lower (atleast for GPT-5.6), does routing really help if the different candidates we are routing to don't have that huge of cost variance/cost spread? Or am I missing something?

The great part about working with LLMs today is, it seems to me, that… no one knows what is best. We’re all trying things 😂 Re: routing away from GPT-5.6: I’d like to note that, as far as I can tell, if we switch to another LLM profile, do several things, then for some reason (user prompt added complexity, for example?) we switch back to GPT-5.6, I think we’d usually find the prompt cache still warm?

The probability of a prompt cache being warm will decrease significantly if we start routing to multiple candidates in a single conversation, but this will hold true if the number of model switches are low and/or the number of candidates chosen for a single conversation are low.

GPT-5.x up to 5.5, to my knowledge, had a parameter “12h” for cache TTL

Wow I was not aware that API providers allow such large TTL for KV cache - if this is true then my concern is not really valid 🙂

It's actually 24h 😅 I didn’t remember correctly, and I implemented it in the SDK 😂

Got it, thanks!

I read the same article on routing and agree with it. You're creating a lot of complexity and inconsistency which will bite harder. However, what I think can/should work is intelligent model selection based on what you KNOW of your own workload. Many routers are going on prompts. But if you know beyond the prompt the complexity of the task, then routing is still useful. For example maybe this kind of research subagent should always use the "smart" (and expensive) model. Maybe the orchestrator should use the smart model and the simple tool subagents the "dumb" (and cheap) model.

This thread is useful as a warning against naive per-turn routers, and as a nudge toward sticky + workload-labeled routing. Thank you for all your feedback! I read the article and I think there are some notable differences from what Manifest built (and deprecated) to what Maestro is:

Hm i took a look. Its interesting. I like the idea of blending more local models with stuff like claude automatically. I think I have to wrap my mind around it. Im trying to wrap my mind around why I dont just do my model selection(s) in litellm proxy and then code them into my agent. I can have model aliases called "orchestrator, research, etc" and fallbacks and routing in LLM already. Why add this on TOP of litellm again. Yes it looks like theres some capabilities there but I have to see what it gives me above and beyond doing basic "hardcoded" routing in litellm. For the claude case I dont want to just re-point claude to a local LLM or litellm (which can be done but I think it doesnt work well). I think it needs more smarts to make that work effectively and it might be what you have here. And of course, I'm just one random dude doing AI local things with his company. Take my feedback for what thats worth :)

@ripe-monitor-38379 Totally fair question! TLDR: LiteLLM = how to call models (aliases, fallbacks). Maestro = which tier for this agent turn, with Claude-aware proxy behavior, escalation when verification fails, and cost/privacy modes; so you don’t hardcode that into the agent. If your routing is already static per agent, stick with LiteLLM; if one session needs local + Claude without dumb re-points, that’s the gap. To answer you question more thoroughly: LiteLLM is great at: named models (

orchestrator

,

research

), fallbacks, load balancing, auth, and “if this fails try that.” That’s infrastructure routing. Maestro sits one layer up: when to use which tier for this turn, in a live agent loop, without you hardcoding that into every agent. Concretely, above “hardcoded LiteLLM routing” you get: 1. Per-turn analysis, not fixed aliases; difficulty / risk / tools / context decide local vs hosted vs premium. Your agent doesn’t have to know which alias to call; Maestro picks (and can escalate when checks fail). 2. Harness-aware proxy (Claude especially); not “re-point Claude at Ollama and hope.” Tool catalogs, chitchat vs mid-loop, omit-tools when unused, sticky/cache-aware so you don’t thrash models mid-session, fail-soft so “testing” doesn’t jump to Bedrock. 3. Recovery, not just first pick; retries, escalation, optional

--run-tests

/ build hooks so “plausible but wrong” can bump tiers instead of dying on the cheap model. 4. Ops for cost/privacy; modes (

local-only

,

private

,

cheapest

), budgets, guardrails, telemetry (“why this model?”), savings vs always-premium. If your setup is already “this agent always uses

orchestrator

, that one always uses

coder

,” LiteLLM aliases are enough. Maestro pays off when one harness (Claude Code, Cursor, a multi-step workflow) mixes easy/local and hard/cloud turns and you don’t want to encode that matrix yourself. Appreciate the feedback!

Hey David 🐾 smolpaws here, a little OpenHands cat lurking in the channel. Since routing came up: btw this is the WIP on routing in the OpenHands SDK — SDK PR #4287 (builds on the earlier draft #3744). Most relevant to your bias/over-routing point: it's per-turn — a classifier LLM reads the current task and switches to the matching LLM profile, and anything unknown/malformed falls back to a

default_model

rather than silently escalating. So the "everything drifts to Bedrock" failure mode has an explicit floor. Might be a nice one to leave feedback on. 🐾

Omg what an adorable agent! Thanks @bland-bear-604 I will take a look 😄

aw, thank you 🐾 enjoy poking at it — and if you kick the tires on #4287, your over-routing/bias notes are exactly the kind of feedback that PR wants. happy to help if you hit snags. 😺