Hi! I’ve been using Promptfoo for a while to create agents and evaluate models. Now, I’m looking to create an agent that calls some tools from a remote MCP in order to interpret the results and produce a specific output. I have configured my provider as follows: - id: anthropic:messages:claude-3-7-sonnet-20250219 label: claude-3-7-sonnet config: temperature: 0.5 max_tokens: 7000 mcp: enabled: true server: url: verbose: true debug: true However, when I create tests, the output always only shows the first tool call: {"type":"tool_use","id":"toolu_01LFZRH5cb34ZZDu6SF3jHES","name":"getBrandSettings","input":{}} Is there any configuration that allows the LLM to execute or plan multiple tool uses inside Promptfoo? I have searched a bit about this and everyone talks about doing a wrapper, but I would like to know if there's an alternative

hi, which version of promptfoo are you using?

Hii, I am using the last version available, 0.119.14

ok i think they have fixed somethin on the latest version about mcp tool calling i also interested in your scenario to test my system what kind of wrapper they sugget to you?

I am creating my own client (in Python) connecting with MCP and a loop to execute all the neccesary tools, returning the final output to evaluate with promptfoo. But I would like to know if there is an easier way of evaluating this 🙏

oh damn, we are in the same situation if i will find something useful i will let you know 🦾

Thanks Matteo! If I find anything I’ll let you know too 🦾

Hey, the official recommended way to evaluate tool call results is indeed to implement a loop with a custom provider. If you don't want to implement your own, promptfoo has one that you can just plug your MCP server into: https://github.com/promptfoo/mcp-agent-provider

Heya! Is there a way to run promptfoo eval with strict filters? For now, it seems you can only filter tests, not scenarios: https://github.com/promptfoo/promptfoo/issues/1061 But also it seems that all scenarios are run, even if the filter doesn't match, which can sometimes be a costly accident. Any way people have worked around this?