I saw these two stories about claude changing billing behavior based on the content of the messages for HERMES.md and mentions of OpenClaw. I wonder if there is any automated testing we could or should do to catch this kind of behavior and flag it

The sad thing is that Anthropic admitted it was a bug and refused to refund the affected users. 😞

Pretty wild that they have specific terms they are flagging in the prompt

I wonder if there is any automated testing we could or should do to catch this kind of behavior and flag it

What do you mean? We cannot use OpenHands with a Claude subscription because of Anthropic's policy (unlike OpenAI). We use it with an API key, i.e., we pay per token. Or do you mean a general way to verify that your Claude subscription doesn't silently change its billing method from one day to the next?

Something like Theo's test with an array of terms we can run regularly. Either through OpenHands or as a standalone test suite. With the Agent ACP support a user would be able to use Claude subscription (I think?) in OpenHands GUI so it would make sense to understand if there are patterns that would eat up credit.

ah ok with the ACP agent protocol make sense