https://openhands-ai.slack.com/archives/C06U8UTKSAD/p1763586013818879 Following up on this. I have been debugging on my end to find what is blocking the payloads. But in the meantime, I'd like to understand the token tracking. When I see

Tokens: ↑ input 18.27K • cache hit 0.00% • ↓ output 175 • $ 0.00

in the cli after a message. Is this actually saying I'm making an 18,270 token request to the model? Or is the actual request some subset of these tokens? Or am I misunderstanding this entirely? With the default tool calling prompt, that input token count balloons beyond the context length I've set for the model after only a few messages, and it is somehow able to continue giving output when I use a locally hosted model. But for a self hosted model on another server, I'm running into issues even when I allow massive payloads. Is that input token count real? And if so how do I make it not increase by 6k tokens on every message/tool call?

That input/output tokens count is not so useful for debugging, sorry 😓 because it keeps accumulating every request

It actually can be useful for debugging when working with vllm/sglang endpoints. Since they are designed to serve many users concurrently, they don't hold the kv-cache, they take the entire context on each request. What I don't know is if the entire chat is in fact the context getting sent.

Ah OK. But sorry, maybe I wasn’t clear, so we get the usage from litellm, which gets it from the model provider, each LLM call: https://github.com/OpenHands/software-agent-sdk/blob/8e296334db0998d34ba9ae0a34a13312df72812a/openhands-sdk/openhands/sdk/llm/utils/telemetry.py#L149 That’s per request, the entire context actually sent. Then we add it up: https://github.com/OpenHands/software-agent-sdk/blob/8e296334db0998d34ba9ae0a34a13312df72812a/openhands-sdk/openhands/sdk/llm/utils/metrics.py#L202 So what is displayed is accumulated context, as best as we can tell (from litellm), over all requests so far in the conversation.