https://promptfoo.dev/ logo
Does promptfoo support agent testing???
# ❓|questions
s
I have a requirement where I have a prompt.. LLM will get the tool information based on the prompt and then execute the tool from MCP server and then send the tool response to the LLM so that LLM will process and then based on the requirement it call another tool and execute the second tool by the MCP server... and the workflow continues until the LLM receives finish_reason as stop. Do we have support for this kind of agent testing by promptfoo..
w
I'm doing this myself using a custom JavaScript provider that runs my agent
s
Hi @Waz can you give some sample code for reference?? I have custom agents build using Theia IDE. I want to evalluate the custom agents using promptfoo.
t
@[Promptfoo] Faizan
u
Hey @saraswathirekhala_44593, some more context could be helpful for better advice, but you could get started by looking at the python provider documentation (https://www.promptfoo.dev/docs/providers/python/), or perhaps the langchain example https://github.com/promptfoo/promptfoo/tree/main/examples/redteam-langchain for some code samples. I could provide some better guidance if you let me know what language / library you're using to develop & run the agent
s
Hi @[Promptfoo] Faizan , I'm trying to evaluate agent workflow.. I'm using provider as below: providers: [ { id: 'openai:chat:gpt-4.1', config: { mcp: { enabled: true, server: { url: 'http://localhost:8091/sse', name: 'local-server', }, }, }, }, ] and for the given prompt this provider only returns me the tool information.. it doesnot execute the tool.. My given prompt depends on more than one tool execution example..."Check if Contact page exists if not present create a fresh page".. for this prompt check_page_exists tool and create_page should be called based on the check_page_exists tool response. I want a provider similar to openai:responses model... based on the prompt it will fetch the tools and execute and return the response to the host... I'm looking some thing like this https://github.com/promptfoo/promptfoo/blob/main/examples/openai-mcp/promptfooconfig.approval.yaml where responses model is getting tool info and executing the tool from the global mcp server and returning the response back to the client.. but the mcp server should be available publicly over the internet this is the limitation for responses model.. I want the same for the remote mcp server and that too multi-turn kind of conversation..
u
Hey @saraswathirekhala_44593 - sorry for the delay. In that case, you're probably best serviced by using a custom javascript provider. Here is an example agent js provider that you can just plug your mcp server into- https://github.com/promptfoo/mcp-agent-provider.