Hi again. always on the condensation(compact) strategies bench. I tried running gpt-6 Luna on 10 hard tasks from swe bench PRO v2. I compared the baseline (uncapped gpt luna 6) with two configs as input limit size : 64K and 32K. my goal is to track the cost and pass rate.
The baseline solved 8/10 task for $0.99 ($0.12 per solve). Conversation window at 64K solves 9/10 tasks for $0.89 ($0.10 per solve). LLM summary solved too 9/10 for $1.03($0.11 per solve).
I share also with you the full paper for more details( it's still unpublished).
I would like to hear your feedback, questions or recommendations...