https://promptfoo.dev/ logo
How to test the output returned by an MCP Tool?
# ❓|questions
z
Hello! I have a local MCP server developed using FastMCP. What I want is to check if a user prompt is causing the execution of a tool and then I want to do some asserts on the output of the MCP tool call. How can I do that locally?
s
hi @zealous_dolphin_81506 did u find some solution... I'm trying the same local MCP server and assert the response but getting "Error retrieving tool list from MCP server" this error. Did u get the tools list??
u
What client/LLM are you using as the agent for the MCP server?
s
hi @User I have some sample MCP server built: Pls find below code. from mcp.server import FastMCP mcp = FastMCP(name="wm-app-mcp-server") @mcp.tool() def add_numbers(a: int, b: int) -> int: return a + b def main(): mcp.settings.host = "0.0.0.0" mcp.settings.port = 8090 mcp.settings.debug = True mcp.run(transport="sse") if __name__ == "__main__": main() And I want to check from promptfoo whether the tools are getting called or not in the response. Pls find promfoo provider info.. providers: # Provider with no approval required - id: openai:responses:gpt-4.1-2025-04-14 label: 'WM MCP Server' config: tools: - type: mcp server_label: uvicorn server_url: http://localhost:8090/sse require_approval: never max_output_tokens: 1000 temperature: 0.2 instructions: 'You are an assistant. Use the available MCP tools to search for information.' debug: true I'm getting the below error when trying to eval the promptfoo yaml file. API error: 424 Failed Dependency {"error":{"message":"Error retrieving tool list from MCP server: 'weather'. Http status code: 424 (Failed Dependency)","type":"external_connector_error","param":"tools","code":"http_error"}} When I checked the server logs I donot see any request coming to the MCP server. I'm trying the same below example given in github but with my local MCP server instead of deepwiki given in the example. https://github.com/promptfoo/promptfoo/blob/main/examples/openai-mcp/promptfooconfig.approval.yaml