https://docs.all-hands.dev/ logo
Join Slack
Powered by
I have a slight question. A couple of monhs ago I ...
# questions
c
I have a slight question. A couple of monhs ago I made a modification to the MicroAgents called selective Context Activation where only relevant parts of agents.md are given to subagents. I've done tests with roughly the accuracy (1% difference in accuracy, roughly 6% token reduction). This is my first time contributing so I wanted to get some guidance on this before I created a pull request.
These were some of my results
c
Hey Jimmy! Sounds interesting I'm just hoping that you tested the current version of OpenHands, V1. In V1, we have renamed microagents to “skills”, as the industry has moved to that name
c
@calm-artist-7206 Looking at it now, no unfortunately I appear to not have. Now granted, I'm sure there is still some sort of microagent spawning in the repository but it has been so long since I have revisited this repo that it has probably moved around unfortunately. I might look at the project board to see what else was being planned.
I somewhat anticipated this when I was researching, however, most AGENTS.md still seem to suffer from some bloat so it still might be useful, so it might still have some merit that I can look into.
c
Please don’t get me wrong, I don’t doubt it could be interesting and useful! I just suggest to maybe try it on agent-sdk (OpenHands/software-agent-sdk repo), as that is currently the openhands agent With its AGENTS.md and other permanent skills, and other Skills in AgentSkills.io format
c
@calm-artist-7206 Yep ahaha I figured and have been working on that repo after finding out that none of the spawning logic was on the main repo
Are there any benchmarks you know of that I could use to test accuracy? I originally used AGENTBENCH, however, that mainly is for single agent as opposed to spawning subagents
c
cc: @millions-crayon-32880 @modern-fireman-99755
👍 1
m
Hey @curved-kilobyte-13755 for benchmarking we are using the OpenHands Index the benchmarks are set up in the OpenHands/benchmarks to run with the sdk
👍 1
The full data of the index can be found here: https://github.com/OpenHands/openhands-index-results/tree/main/ Running the full index is costly, so my advice would be to compare a very small split of, say SWE-Bench, and compare.
c
@millions-crayon-32880 Hey Juan. I'll likely be testing with an open weight models on my DGX Spark. Probably either Gemma4 or Nemotron 120B. Last time I ran something like this it did take over 6 hours and roughly like $50 of OpenAI tokens, but now with 128GB of unified memory/different hardware, i'll see how much I can run. Would it be of any help to submit results (baseline results no matter if my method works or not) even if not all tasks were ran? Or would you only take results if I ran full suite
And if so, is there any chance I would be able to get a RUNTIME_API_KEY
m
@curved-kilobyte-13755 the index only contains evals done from scratch by OpenHands. We are not sharing api keys. Taking a second look, before any benchmarks, how does Selective Context Activation differ from the Skills feature that Engel mentioned earlier?
c
@millions-crayon-32880 So it isn't technically about skills, rather about cutting out unnecessary things from the AGENTS.md. Rarely does a subagent need to know every single thing mentioned in AGENTS.md. What it does it gives the orchestrator a list of subsections (along side an always active header) of what information should and shouldn't be given to the subagent to save on context
From my understanding, no matter what happens with an agent, AGENTS.md is always included for an agent no matter its task
Selective Context Activation essentially cleansd up what isnt already included in formal skills or excess from redundant AGENTS.md files. Granted my research was done back in March when this feature was made
For instance on the tests im running, all of these were found from repository specific AGENTS.md, granted this isnt a great example as all sections were activated but I have to run through an x86 emulator which slows down the tests very much
m
Oh I see, so Selective Context Activation prunes contents of AGENTS.md and feeds that to the agent depending on the task.
c
Yep!
Sometimes the excess content leads to the LLM exploring more for no reason or following rules TOO much
m
The benchmarks I mentioned so far do not consider AGENTS.md
What signal did you got on AGENTBENCH?
same result at lower cost? better result at same cost?
c
Out of 138 tests, only one less correct but like $0.10 less per test
👀 2
But this heavily differs based on the model. When I first started this in March, GPT 5.3 got like 30%. In April it somehow got 68% which was frustrating to say the least
m
Can you share a table of what you have already benchmarked? Model, Score with SCA, Score without SCA, Cost with SCA, Cost without SCA
I'm trying to see what percent of the overall cost was reduced.
c
@millions-crayon-32880 I can do you one better because this was originally supposed to be a paper. My main gripe with the paper was the original AGENTBENCH token tracking through litellm and vllm had an issue with counting tokens so I manually tracked the deltas through cost
Formatting_Instructions_For_NeurIPS_2025-2.pdf
m
Evaluated on AGENTBENCH (138 instances, 12 repositories),
SCA achieves a 69.6% task success rate compared to 71.0% for full AGENTS.md
loading and 70.3% for no context, while reducing prompt tokens by 2.9%, steps by
8.4%, and total API cost by 22.4%.
that's interesting
c
Is it worth a pull request as a beta feature or a setting to implement for token conscious people?
m
@curved-kilobyte-13755 if you create a PR for software-agent-sdk I'll run some manual tests.
c
@millions-crayon-32880 Created a PR. I didn't create anything on the front end because I was unsure of how/what direction this would be going in as a beta feature or what + I'm mainly an AI engineer/research and my front end would probably not be up to par unless I used heavy amounts of AI. github.com/OpenHands/software-agent-sdk/pull/3890
m
@curved-kilobyte-13755 thank you! I've assigned myself for review, should have some feedback next week
✅ 1
c
@millions-crayon-32880 thank u for the feedback much appreciated i’ll try to get everything sorted out today
@millions-crayon-32880 let me know if there is anything else I can do/run/test to help the process
m
@curved-kilobyte-13755 thanks, will review it!
✅ 1
c
bump
2 Views
Open in Slack
PreviousNext
Powered by