https://docs.all-hands.dev/ logo
Join Slack
Powered by
Been spending some time looking through the Claude...
# proj-agent
w
Been spending some time looking through the Claude Code source to see what tricks they use to handle context. There's a lot. Details in the 🧵.
The first layer is ensuring that events don't get too big. CC truncates tool calls that are too large, just like OH, but they have different limits per-tool where the SDK relies on 
maybe_truncate
. Also some paths where CC writes huge events to disk and modifies the event to reference that file instead.
The next layer is cache-specific behavior. Big benefit of Anthropic controlling the model and the agent harness is that they can hyper-optimize for their cache behavior. One trick I noticed is when there's a big time gap between user messages they'll go back through and "mask" tool calls to save on context, since the cache is already cold.
The last layer is actual condensation/compaction. There's at least two strategies I could find for CC: there's an LLM-based compaction strategy similar to our context condenser, and there's an experimental memory-based compaction strategy that uses a post-turn hook to summarize the conversation while it's still happening.
The LLM-based compaction has a few optimizations that are worth noting. First, they have some structure to the prompt, just like OH does, although the categories are a little different. Second, they re-use the cache in a forked agent with 
max_turns: 1
to generate the summary. We experimented with this in v0 and didn't find much benefit (see here: https://github.com/OpenHands/OpenHands/pull/7893), might be worth revisiting in v1.
The last trick I saw that was kind of interesting was how they re-prompt the agent post-compaction. Once a summary has been generated, they re-build part of the message history by re-loading recently accessed files and repeating the most recent tool calls. Interestingly, the OH condenser does this implicitly by only summarizing the first half of all messages.
There's more of course, but these are the main ideas worth noodling on for us (except the cache-specific behavior, that's probably not worth the time/complexity for us)
nod 1
c
there's an experimental memory-based compaction strategy that uses a post-turn hook to summarize the conversation while it's still happening.
This sounds very interesting! 🤔 I don't know if it's post-turn, as in, when the successive "agent loop" of tool calls finished, or as in, after each tool call. I am still to check, guessing the second. "Live summary" I was thinking, on a completely different matter, a live summary would be great to have: as conversations grow longer, and LLMs get better, I seriously doubt the user wants or needs to see every tool call; some kind of summary of what the agent did might be a nicer experience
w
Do we have ways to fork a conversation using a hook? There's a lot we could do with summarizing tool loops by just spinning up a quick agent with the same context (to maximize cache hits) and exposing the result so SDK consumers (like the CLI) can render it how they choose.
c
🤷 the hook can be post-tool call, but I don't know if it gets the conversation...
4 Views
Open in Slack
PreviousNext
Powered by