https://www.boundaryml.com logo
Powered by
Tips on reducing BamlValidationErrors
# help
s
I'm building a coding agent with some standard tools. The model itself is round robin between Sonnet 4.5 and GPT 5. I'm running into issues where BamlValidationErrors are coming up because the output doesn't conform to any of the tool structures. However, I'm not able to reason the core reason and whether I'm doing something really silly. Adding snippets in the thread.
Snippet from logs
Copy code
user: build an app that says "hello"

Raw Response: I’ll inspect the project, add a minimal screen that displays “hello,” and verify the Expo server builds cleanly.
[chat] Final validation error raw output: I’ll inspect the project, add a minimal screen that displays “hello,” and verify the Expo server builds cleanly.
[chat] State at error (last 3 messages): [
  {
    "role": "user",
    "message": "build an app that says \"hello\""
  }
]
[chat] Coding agent encountered a BAML error: BamlValidationError: Failed to parse LLM response: Failed to coerce value: <root>: Failed to find any (ListFilesTool | ReadFileTool | WriteFileTool | BashTool | VerifyExpoServerTool | TodoWriteTool | ReplyToUser) @stream.not_null in 7 items
  - <root>: Failed while parsing required fields: missing=2, unparsed=0
    - <root>: Missing required field: action
    - <root>: Missing required field: directoryPath
  - <root>: Failed while parsing required fields: missing=2, unparsed=0
    - <root>: Missing required field: action
    - <root>: Missing required field: filePath
  - <root>: Failed while parsing required fields: missing=3, unparsed=0
    - <root>: Missing required field: action
    - <root>: Missing required field: filePath
    - <root>: Missing required field: content
  - <root>: Failed while parsing required fields: missing=2, unparsed=0
    - <root>: Missing required field: action
    - <root>: Missing required field: command
  - <root>: Failed while parsing required fields: missing=1, unparsed=0
    - <root>: Missing required field: action
  - <root>: Failed while parsing required fields: missing=1, unparsed=0
    - <root>: Missing required field: action
  - <root>: Failed while parsing required fields: missing=2, unparsed=0
    - <root>: Missing required field: action
    - <root>: Missing required field: message
@baml-sam
Let me know what else I could provide here! Thanks for your time
b
What's in 
AgentTools | ReplyToUser
?
s
b
What kind of responses are you getting back?
s
Copy code
user: build an app that says "hello"

Raw Response: I’ll inspect the project, add a minimal screen that displays “hello,” and verify the Expo server builds cleanly.
[chat] BAML error raw output: I’ll inspect the project, add a minimal screen that displays “hello,” and verify the Expo server builds cleanly.
[chat] Chat failed: Coding agent encountered a BAML error
I think I'm getting just a single string as shown in the raw response, which is not conforming with tool structure. I'm wondering if this could be due to any reasoning/thinking capabilities
b
The fact that you're getting a single string sounds right to me. I'm not sure if I would blame that on reasoning/thinking, but I wouldn't rule it out
Hmm...
Your log format doesn't look like the baml log format. Did you set BAML_LOG to something else?
s
Alright I'm doing some tests with the BAML unit test setup, will let you know if I see anything more
Weird that it has multiple tool calls in the raw llm response...
^ that was with Sonnet 4.5, with GPT5 I get the raw llm response to be
Copy code
I’ll set up a minimal Expo app and display “hello” on the main screen. Proceeding to inspect the repo, scaffold (if needed), and implement the screen.
I'll try playing around with the thinking field a bit
b
Try adding an example of a tool call in your "Good example" section
So like the actual JSONish response that you want back
sonnet 4.5 is clearly biasing towards XML for some reason which is interesting...
s
I'll try that, thanks
So I added a simple "Your tool choice answer MUST be in JSON format. Do NOT give tool call choices in any other format." and now it's working so much better. Clearly I forgot to specify that explicitly so perhaps it was assuming. I was working with GLM before Sonnet/GPT5 and it was fine there, so maybe GLM is biased towards outputting in JSON by default
b
Yeah, I think it's also just a consequence of what "tool call" is actually associated with on the internet.
Long-term we want to support more output formats so you don't have to do silly stuff like this, but for now it's still needed
s
Thanks for the help!
PreviousNext
Powered by