https://www.boundaryml.com logo
Powered by
Anthropic stop_sequences sometimes get ignored
# help
a
Every once in a while, I've been getting a request where configured 
stop_sequences
get ignored by the provider, the LLM would produce the stop sequence and keep going. I was thinking this is probably something at the provider level, but this did happen with both Anthropic and Bedrock providers (with Sonnet 4.5), and also I can't remember this happening before I upgraded to BAML 0.214.0 (was on 0.209.0 previously). So now I'm really unsure what the cause of this could be and whether its possible that BAML could be a factor. The configured 
stop_sequences
work 99% of the time, and I'm not able to consistently reproduce the issue, which is why this is so strange.
v
Hmm, we haven’t really made client changes to baml in quite a bit so that seems suspicious. I wonder if the model provides the correct stop_sequence?
Or it could be a greedy tokenization: where what they do is convert your stop sequence to tokens, and while it does appear like the model produces the string that looks like your stop sequence, it isn’t the same tokens (since a string can be tokenized in multiple ways)
a
Yeah it definitely provides the same sequence. In addition I’ve had it repeat the exact same response which produced the stop skip and failed to replicate, which should in theory have the same token decomposition, which suggests that’s probably not the issue but not 100% on that
v
do you know if anthropic is actually sending the right stop reason? the new studio shows the full http request / response so confirming this should be easy! https://cdn.discordapp.com/attachments/1448940958354771998/1449218235110002818/Screenshot_2025-12-12_at_5.55.30_PM.png?ex=693e1928&is=693cc7a8&hm=be83fdeb83166d491ac7ab920100edef0f570a85a4ca96254bcd7acd1b095bfa&
a
PreviousNext
Powered by