Hey! 🙂 Lately, I’ve been working a lot on token usage, costs, and cache optimization. I’ve wrapped up my notes into a short article where I try to explain, from a (very) simple mathematical point of view, what is going on in agentic conversations. I also ran some small experiments using the OpenHands index trajectories. I hope this is interesting for some of you. Let me know what you think! Article here.

Very cool, you should also note how long the prompt caching lasts. Didn't Anthropic reduce the time from 1 hour to 5 minutes.

According to the latest docs, you can still have a one-hour cache if you pay more (ref). However, every time you hit the cache, the timer resets. This means that if you make at least two API calls less than 5 minutes apart, you are good. This is generally enough for most agentic workflows, but for coding, it can be a problem, as running tests or scripts can sometimes take more than 5 minutes. PS: I don't know if the Opus trajectories were using the 1h or the 5min cache.

Good stuff, it makes me think there are probably better ways to auto manage the context windows. Right now it's on me to decide when to clear the memory because I shifted tasks or we wait to summarize until the window is pretty full. Hmm

Awesome article!

Great write up! Thank you for sharing!