https://docs.all-hands.dev/ logo
Join Slack
Powered by
Hey everyone, I've opened two PRs to address the ...
# proj-agent
m
Hey everyone, I've opened two PRs to address the OpenAI caching issue: 1. fix(llm): pin prompt_cache_key to conversation id 2. fix(llm): preserve Responses function_call.id for byte-identical repl… Note: PR #2 merges into PR #1, and PR #1 merges into 
main
. Both PRs include an 
eval-50
to ensure no regressions. Additionally, I ran a combined 
eval-50
with both fixes together to confirm they resolve the caching problem: https://openhands-ai.slack.com/archives/C09QGUDQVTL/p1776846701167599 <- analysis in 🧵
👀 1
Aggregate results for 50 instances.
Copy code
Metric                  Value                                                                                         
  ----------------------  --------------------------------
  Cache-hit ratio         96.3%                                                                                         
  Min per-instance hit    92.8%
  Max per-instance hit    98.1%                                                                                         
  Instances <50% hit      0                                       
  Mean iterations/inst    34.7 (p95 51)
  Errors / retries        0 / 0                                                                                         
  Total cost              $32.90 ($0.87 per resolved)
  Pass@1 (50 submitted)   76% (38/50)
Comapring against the eval we have in the openhands index
Copy code
pre-fix (d94cdf8)   post-fix #2916 (f15c68f)
  Cache-hit ratio         63.5%               96.3%                                                                     
  Min / p50 / max         31.7% / 61.9% / 89.7%   92.8% / 96.1% / 98.1%
  Instances >=90% hit     0 / 467             50 / 50                                                                   
  Cost per instance       ~$1.45              ~$0.66
c
I set off my agent to investigate what about switch LLM and fork conversation. It will post two issues for us to see.
m
Oh thanks 😄 i forgot about fork and switch… is there a way to have everything in sync such that we can change just in a place?
c
For LLMs, that place feels like the LLMRegistry. I don't think we currently pass through it quite... as we could 🤔
TBH I think maybe we will need to investigate and rethink some, considering also the additions since the last deeper architectural work; sub-agents; profiles; acp; and we do have some model_copy around... 🤔 Because indeed you're right, we probably should have one place (or two at most if it makes sense), otherwise things drift and bugs crop up 🤷
I think my ideal end goal would be to have no mutation of LLM, conversation, and agent, except in very few and well defined places, if that makes sense.
m
Make sense. I'm investigating. I will take care of the two issues that you opened 😉
gratitude thank you 1
l
Thanks a lot for this @modern-fireman-99755! If it’s already working in the simple case of running the openhands index, could you kick off runs to re-test gpt-5.4 on all benchmarks?
👍 1
(and thanks @calm-artist-7206 for the reviews!)
🙌 1
6 Views
Open in Slack
PreviousNext
Powered by