1. providers: - id: openai:chat:gpt-4 config: mcp: enabled: true server: name: my-mcp-server url: '' it gives response as json like this ## [{"id":"call_X4129obXuAlKku02qOKFWk6d","type":"function","function":{"name":"namespaces_list","arguments":"{}"}}] instead of getting actual data. 2. providers: - id: openai:responses:gpt-4.1-2025-04-14 config: tools: - type: mcp server_label: my-mcp-server server_url: require_approval: never allowed_tools: ['namespaces_list'] max_output_tokens: 1500 temperature: 0.3 instructions: 'You are a helpful research assistant. Use the available MCP tools to search for accurate information about repositories and provide comprehensive answers.' i get this : API error: 424 Failed Dependency {"error":{"message":"Error retrieving tool list from MCP server: 'my-mcp-server'. Http status code: 424 (Failed Dependency)","type":"external_connector_error","param":"tools","code":"http_error"}} i am using latest version of promptfoo can anybody help

@[Promptfoo] Faizan pls?

Hey, what I'd recommend if you're trying to test the results, that you use a custom provider. We have a simple template for you to modify here: https://github.com/promptfoo/mcp-agent-provider/. There's instructions to just modify the config to allow you to specify your mcp server. Let me know how that goes or if you have any questions!

@[Promptfoo] Faizan i want to test a remote mcp server with headers taking some authentication value like x-api-key, Authorisation ... Thats why i had tried the above options, i saw the code base you have shared, does it allow remote mcp server? i cant see any documentation for the same.

Hey @suraj_43399, the example I shared does support remote servers. Take a look at our docs here: https://www.promptfoo.dev/docs/integrations/mcp/. Just modify the promptfooconfig.yaml in that repo with your config and it should work

i tried @[Promptfoo] Faizan Evaluating [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0% | 0/1 | Failed to connect to MCP server: The "file" argument must be of type string. Received undefined Failed to connect to MCP server undefined: TypeError [ERR_INVALID_ARG_TYPE]: The "file" argument must be of type string. Received undefined at normalizeSpawnArguments (node:child_process:548:3) at Object.spawn (node:child_process:755:13) at spawn (/Users/surajprabakark/promptfoo/mcp-agent-provider/node_modules/cross-spawn/index.js:12:24) at /Users/surajprabakark/promptfoo/mcp-agent-provider/node_modules/@modelcontextprotocol/sdk/dist/cjs/client/stdio.js:74:55 at new Promise () at StdioClientTransport.start (/Users/surajprabakark/promptfoo/mcp-agent-provider/node_modules/@modelcontextprotocol/sdk/dist/cjs/client/stdio.js:72:16) at Client.connect (/Users/surajprabakark/promptfoo/mcp-agent-provider/node_modules/@modelcontextprotocol/sdk/dist/cjs/shared/protocol.js:99:31) at Client.connect (/Users/surajprabakark/promptfoo/mcp-agent-provider/node_modules/@modelcontextprotocol/sdk/dist/cjs/client/index.js:65:21) at MCPClient.connect (/Users/surajprabakark/promptfoo/mcp-agent-provider/src/mcp-client.js:34:25) at OpenAIAgentProvider.initialize (/Users/surajprabakark/promptfoo/mcp-agent-provider/src/openai-agent-provider.js:32:22) { code: 'ERR_INVALID_ARG_TYPE' } Evaluating [████████████████████████████████████████] 100% | 1/1 | file://./src/openai-agent-provider.js "{{prompt}}" prompt=Get the remote server is not connected

Hey @suraj_43399, thanks for letting me know- I think this repo was a bit outdated. I just pushed an update that should support your use case, please give it another try 🙂

@[Promptfoo] Faizan , thanks, it is working after the update. is this something under development ? how can we use it like other providers as in the documentation.

Hey @suraj_43399, promptfoo users typically end up using custom providers like this one when they have complex use cases, so moving forward with your testing with this provider is totally fine. I do have an idea of adding this as a simple provider in a future release though.

Thanks @[Promptfoo] Faizan

hey @suraj_43399 , the only thing I would change is that you should add the tools your mcp server has to the redteam.purpose

Thanks @[Promptfoo] Faizan , i will try them