https://www.boundaryml.com logo
Powered by
Best practice for rendering tool call history in m...
# help
v
I'm building a multi-turn agent loop where the BAML function is called repeatedly until a final answer is produced. The conversation history (including past tool calls and their results) is passed as a 
messages
parameter and rendered in the prompt template.
## Setup - BAML function receives 
messages: Message[]
containing 
user
, 
assistant
, and 
tool
role messages - Each 
assistant
message may contain 
tool_calls
(id, name, arguments) - Each 
tool
message contains the result for a specific 
tool_call_id
- The function is called in a loop: LLM returns a tool call → executor runs it → results are appended to messages → function is called again
## Current template approach I render tool calls and their results together inside the assistant block, and skip standalone 
tool
messages:
Copy code
jinja
{%- for msg in messages %}

{%- if msg.role == "user" %}
{{ _.role("user") }}
{{ msg.content }}

{%- elif msg.role == "assistant" %}
{%- if msg.tool_calls %}
{{ _.role("assistant") }}
{%- for tc in msg.tool_calls %}
→ tool_call id={{ tc.id[-6:] }} name={{ tc.fn_info.name }}
→ params: {{ tc.fn_info.arguments }}
{%- for result_msg in messages %}
{%- if result_msg.role == "tool" and result_msg.tool_call_id == tc.id %}
← result id={{ tc.id[-6:] }}:
{{ result_msg.content }}
{%- endif %}
{%- endfor %}
{%- endfor %}
{%- else %}
{{ _.role("assistant") }}
{{ msg.content }}
{%- endif %}

{%- elif msg.role == "tool" %}
{# already rendered inside the assistant block above #}

{%- endif %}
{%- endfor %}
## The problem After tool execution, the last two entries in 
messages
are 
assistant
(tool_calls) + 
tool
(result). Since 
tool
messages are skipped in the template (rendered inside the assistant block), the final API message sent to Claude is an 
assistant
message. Claude treats this as a prefill — it sees a completed tool call with its result already present, considers the turn done, and returns near-empty content (a few whitespace tokens → 
end_turn
). This worked fine with Gemini, but broke after switching to the Anthropic provider because Claude interprets trailing assistant messages as prefill.
## What I'm considering 1. Render tool results as 
_.role("user")
 instead of embedding them in the assistant block — this naturally makes the last API message a user message, avoiding the prefill issue. This also aligns with Anthropic's native Messages API where 
tool_result
lives inside user messages. 2. Append a synthetic user message in the application layer before calling the BAML function again — works but feels like a workaround.
## Questions 1. Is there a recommended BAML pattern for rendering multi-turn tool call history that works well across providers (especially Anthropic)? 2. Is rendering tool results under 
_.role("user")
the right approach, or is there a better way to handle this in BAML templates?
a
Hey @vEth - this is super interesting and we notice similar issues. My mind went to similar considerations before reading yours. Did you get any further here?
v
I observed a different form of parsing error and also discovered a bug in BAML. You can check this link. But the most significant adjustment I see right now is changing how the tool is rendered. Although it doesn't require much work, the impact is quite substantial.
s
[EDIT] the tool result is actually sent with _.role("tool") --- I'm using Qwen with the Hermes-format chat template and know that tool results should be used with the 
user
role. Here’s what the tool response looks like from my BAML integration with langchain’s create_agent:
Copy code
<|im_start|>system
You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.
      
Answer in JSON-like string (no double quotes in keys) using this schema. Call a tool only when explicitly needed for the task; otherwise, respond with text-only output:
string or {
  // If you want to use a tool, specify the tool and its arguments here.
  selected_tool: {
    // Add two integers.
    name: "add",
    arguments: {
      // First integer to add
      a: int,
      // Second integer to add
      // Default: 5
      b: int,
    },
  } or {
    // Multiply two integers.
    name: "multiply",
    arguments: {
      // First integer to multiply
      x: int,
      // Second integer to multiply
      // Default: 5
      y: int,
    },
  },
}<|im_end|>
<|im_start|>user
1+2=?<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "add", "arguments": {"a": 1, "b": 2}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
3
</tool_response><|im_end|>
<|im_start|>assistant
It might be worth checking: - What chat template Claude actually uses (if it even exposes one) - Whether there’s any way to inspect it via a self-hosted setup like vLLM - Or alternatively, looking at LangChain’s ChatAnthropic implementation to see how tool results are handled internally https://cdn.discordapp.com/attachments/1471049818108592158/1471465060722278561/image.png?ex=698f0820&is=698db6a0&hm=f63b370fb738011feca2a25693bf34972e9067c4803a752d812b7380566627a0&
v
BAML plans to natively support tool calls, and this approach is worth referencing.
s
Oops, it should be 
_.role("tool")
instead.
Copy code
client<llm> ChatBaml {
  ...
  allowed_roles ["system", "user", "assistant", "tool"]
  ...
}

{{ _.role( "tool" ) }}
3
chat template render this:
Copy code
<|im_start|>user
<tool_response>
3
</tool_response><|im_end|>
2 Views
PreviousNext
Powered by