Speaking of constrained decoding vs. parsing... 👀 https://cdn.discordapp.com/attachments/1119375594984050779/1452820539226980383/Screenshot_2025-12-22_at_4.28.19_PM.png?ex=694b3410&is=6949e290&hm=772260296cbeeaa0a5dcceb1ae724b8d8e3b180d67d0f8da63772ac3a876eb7b&

Interesting! theres's a new dataset! thanks for sharing this. During an open week, we'll give it a shot. My hunch says that most FSM approached (outlines, guidance, openai structured outputs) all perform similarly in what they do: * provide a high level of guarantee on the structure of the output * provide a medium level of guarantee on the semantics of the output I'm always curious to ask our users why they find BAML ends up producing better quality results in their scenarios, and I suspect its because: 1. real world inputs are much more diverse than "known good". So in the scenario when 5-10% of your inputs from users are non-sense the structured output approach ends up limiting the model too much. I've yet to see a benchmark that sees the impact of "close but not right inputs" mixed into the dataset. 2. most folks use a vendor's model, so using generalized FSM techniques aren't possible since the APIs only expose a form of it that maps to a JSON schema, which returns back to problem 1. 3. likely early next year we will just be enabling structured output API access 😉 so it'll be a moot point very soon (since devs should have to choice to use whatever technique comes out and BAML should ideally be agnostic)

Interesting. I use exclusively models that support structured output, and I’ve never run into issues with the schemas I specify. I have seen BAML fail, which is expected given that it doesn’t guarantee a particular structure. The ability to get structured outputs reliably across models--without dealing with vendor-specific quirks or package-specific conversions--is what really makes BAML’s parsing approach shine.