OpenAI vs Anthropic models - tendency to adhere BAML prompts Hey! We have a number of agent loops and simpler BAML calls that we have been stably running agains GPT4.1 models for a while. We are now evaluating these BAML files against Anthropic models and our team is suprised how hard a time both Haiku, Sonnet, and Opus have to align with BAML schemas! For example, where we almost never got BAML parsing errors, we now see them very often across all Anthropic models. Has anyone observed a similar tendency? What have you done about it?

What kind of errors do you run into?

ParsingError { scope: [], reason: "Failed while parsing required fields: missing=2, unparsed=0", causes: [ParsingError { scope: [], reason: "Missing required field: tool_name", causes: [] }, ParsingError { scope: [], reason: "Missing required field: raw_answer", causes: [] }] }] }'

Mostly parsing errors like the above. This is from an agentic loop with at discriminated union type for different tools and a common "tool_name" field (inspired by the examples @Vaibhav (BAML 🐑) shows on the ai-that-works podcast). Again, what puzzles us is that we have never seen these errors with e.g. GPT4.1 and -mini (which are 10 or so months old) and we now see them often across all most recent Anthropic models. Would love to show a concrete benchmark but there is a striking difference in our hands at least. Any idea what might be going on?

@veth0119 - super interesting! Thanks for sharing and your analysis - there is something here to uncover of what actually work with the Claude models when using BAML esp. in an agentic loop.

can you share what raw text you're getting out here from the model?

RAWOUTPUT

→ tool_call id=cal_001 name=create_calendar
→ params: {"summary":"Review contract","description":"review contract","start_time":"2026-02-13 08:30","end_time":"2026-02-13 09:00","reminder_minutes":0}

in an agentic loop what i found is that if you don't use the same format to serialize the previous calls in the messages, it will get confused

I think so too, but it wastes a lot of tokens, which is why I’ve been reluctant to make changes.