https://www.boundaryml.com logo
Powered by
BAML Dynamic Schema: Silent Fallback Parsing Cause...
# help
v
## Summary When using BAML with 
@@dynamic
classes in a union return type, malformed LLM outputs may be silently parsed as valid (but empty/useless) values instead of throwing 
BamlValidationError
. This can cause infinite loops or unexpected behavior in agent-based architectures.
This is my analysis
v
1 easy workaround:
Copy code
type Tools = ToolCall[] @assert(require_non_empty, {{ this|length > 0}})
function Orchestrator(...) -> Tools | Answer {
    // ...
  }
for the empty tool, thats interesting. i need to check how that can happen
v
The above analysis was derived from Codex, and I had Claude Code format it
Why @@dynamic + Non-Empty Assert Doesn't Work Core Issue When ToolCall is purely @@dynamic (no required fields), BAML's parser will "fallback-parse" any invalid output into a schema-valid result: llmResponse: "hello world" โ†’ [{}] โ† length=1, non-empty assert passes llmResponse: "null" โ†’ [{}] โ† length=1, non-empty assert passes Why This Happens A @@dynamic class has no predefined required fields, so an empty object {} is itself a "valid" ToolCall. Therefore: 1. The parser cannot distinguish between "invalid output" and "valid but empty fields" 2. It tends to produce a "minimal valid structure" [{}] rather than throwing an error 3. [{}] has length 1, so @assert({{ this|length > 0 }}) passes Comparison: With Required Fields When TypeBuilder injects invoke as required: llmResponse: "hello world" โ†’ [] โ† cannot construct valid element, degrades to empty array llmResponse: "[{}]" โ†’ [] โ† {} lacks invoke, discarded Only then does the non-empty assert become effective. TL;DR @@dynamic with no required fields = empty object is valid = fallback to [{}] = bypasses non-empty assert
v
Yea that I agree with is def a problem. Iโ€™ll have to dig into why the parser is doing that! Does your dynamic type have fields in that class? If so we shouldnโ€™t parse an empty struct into it.
v
- ToolCall = { description: string } + @@dynamic๏ผˆNo other constraints๏ผ‰ llmResponse: "hello world" -> ToolCall[] (Possibly [{ "description": "hello world" }]) llmResponse: "null" / "{}" -> ToolCall[] (Commonly known as [])
Any update๏ผŸ
v
hi sorry @vEth been travelling for a second here. I'll have to take a look this later thsi week to figure out why this happening. if you are able to, do you happen to have a minimum repro of what types you're using?
v
I'll give it a try, I'll use a weaker model to see if it's easier to reproduce
v
appreciate it!
v
@Vaibhav (BAML ๐Ÿ‘) Please check
v
thanks for the very easy repro
testing now!
sorry i've been out of town for a bit but back today!
ok so i think there's def a bug re the typebuilder. it shouldn't be parsing when you don't provide all the fields in the string so will try and take that on for friday.
thanks for being super proactive on this!
v
I am very much looking forward to the new baml features, but I am not clear about your roadmap. Although I was informed before, it seems to still be postponed.
v
sadly still a bit postponed ๐Ÿ™ we've been making steady progress! but eng efforts a bit more than we anticipated to give the level of developer experience we are satisfied with
5 Views
PreviousNext
Powered by