In the simple-mcp example, how are the the

tool

and

args

test vars translated to json given the static 'MCP Tool Call Test' prompt? https://github.com/promptfoo/promptfoo/blob/main/examples/simple-mcp/promptfooconfig.yaml

That example is a bit misleading, I should probably update it. The prompt isn’t used when using an mcp as a provider- instead, Promptfoo generates tool calls to try to simulate different attacks, as if a malicious client has connected to an mcp server (Promptfoo acts as the malicious client).

Thanks @[Promptfoo] Faizan. I swapped out the

./example-server.js

mcp server for the Docker MCP Toolkit, but tests fail with "Invalid JSON in prompt. MCP provider expects a JSON payload with tool call information."

prompts:
  - 'MCP Tool Call Test'

providers:
  - id: mcp
    label: 'MCP Security Tester'
    config:
      enabled: true
      servers:
        - name: security-test-server
          command: docker
          args: ['mcp', 'gateway', 'run']
      verbose: true
      debug: true

tests:
  # Test basic file reading
  - vars:
      tool: 'read_file'
      args:
        path: 'package.json'
    assert:
      - type: not-empty
      - type: contains
        value: 'simple-mcp-example'

Whoops, you're right- I misunderstood and thought this was the red team case. In any case, the test cases are definitely a typo and the mcp provider relies on

vars.prompt

to be the test case json. 😅 I'll correct the example, thank you for the question!

Thanks for the clarification. I was also wondering if there's a way to use the nunjucks

dump

filter in the prompt to create the json message to the mcp server, but it doesn't seem to work. From what I can gleam,

toolName

and

toolArgs

are not parsed into the test metadata with this approach.

prompts:
  - '{{ prompt | dump }}'

tests:
  # Test basic file reading
  - vars:
      prompt:
        tool: 'read_file'
        args:
          path: 'package.json'

Ah, yeah, the direct mcp provider was really built for red teaming, so it doesn't follow the usual system for evals. What you're describing makes sense though- I don't have bandwidth to implement that at the moment, but you're welcome to submit a PR (Or, I'll get to it, maybe next week). Shouldn't be too hard, I can also describe how I'd make the change

Makes sense. If you point me in the right direction, I could look into spiking up a PR. thanks again!