I would be grateful for feedback on #6383: Detect and recover from prefix-cache collapse on hosted/self-served LLM backends (per-call telemetry + backoff policy)

> Why this belongs in the SDK, not application code I am not 100% sure about this: • It's the responsibility of the inferencing system to function correctly under pressure, not the responsibility of the client to ◦ detect a problem in the inferencing system and ◦ assume that this problem is because of high load and implement a workaround fix to reduce load but degrade performance • What happens if cache collapses for another reason than high load? This system will activate and slow down agents? I mean if this was as problem we faced with openrouter too we'd definitely want to implement it, but if this is specific to a customer's llm inferencing service...

Thanks very much for looking at it! Responded on the issue itself.

is their llm inferencing service opensource?

I am looking into those questions and options now... Learning a lot about various inference engine settings...

i'm just worried that this failure mode is not universal at all and we're implementing a fix for a client particularity

Exactly the right question. I was able to confirm that this is a common and well understood failure mode across a number of open source inferencing engines. I am now working on getting a better handle on what knobs are offered to address this differently on the inferencing engine side.

Or in litellm, possibly?

On the LiteLLM site it is possible to write admission control plugin that would catch this sort of thing proactively and I have a rough design for it. If there isn't a simple solution on the inferencing side, then it would be good to do that, but I think you'd still want a circuit breaker in the agent to deal with case where something slipped through that defense. That said, I do agree that best solution is on inference side if there is a realistic one. I have some leads on that now and will report back when I can confirm a solution or that no realistic solution exists.

I was thinking at litellm sdk, which has some retries, and other code around the LLM provider calls, but I could be wrong

It has a framework that can help but not a complete solution out of the box via settings alone.