https://promptfoo.dev/ logo
Having errors when using MCP with Gemini 2.5 flash
# ❓|questions
u
I'm running into an issue with promptfoo eval using this config: - id: google:gemini-2.5-flash-preview-05-20 config: mcp: enabled: true server: url: https://mcp.server When I run it, I get this error: Error: Expected one candidate in API response. If I set enabled: false, everything works (just without MCP). Also, if I switch to a different model like chat-gpt-4.1, it works fine too. What's strange is that we're already using this same Gemini 2.5 Flash model with MCP elsewhere, and it's working — just not through promptfoo. Is this a config issue on my end, or something I should report as a bug in promptfoo? Thanks in advance for any help!
I also tried with Vertex using the same model. It returns a different error. API call error: Status INVALID_ARGUMENT, Code 400, Message: Invalid JSON payload received. Unknown name "$schema" at 'tools[0].function_declarations[0].parameters': Cannot find field. Invalid JSON payload received. Unknown name "$schema" at 'tools[0].function_declarations[1].parameters': Cannot find field.
a
TL;DR: Workaround with removing $schema in MCP server. Will look into adding extra parsing in Promptfoo tomorrow. Reproducing the error 2 (Vertex): Using the model with Vertex - the error reproduced by the MCP server specifying the $schema field:
Copy code
jsx
tools: [{
        name: "get_weather",
        description: "Get weather information",
        inputSchema: {
          "$schema": "http://json-schema.org/draft-07/schema#",  // This causes the error!
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "City name"
            }
          },
          "required": ["location"]
        }
      }]
By the [MCP specs](https://modelcontextprotocol.io/specification/2025-03-26/server/tools) the $schema is not specified but rather inferred. In reproducing the error this specifically affected the Vertex model, not chat-gpt-4.1. So, some additional parsing (removing $schema) can be added to
transformMCPToolsToGoogle
in Promptfoo to avoid this issue. Do you know if this MCP or similar is available for me to test? You could standardize the input to align with MCP specs to work for now. Reproducing the error 1 (Google): I reproduced the same
Error: Expected one candidate in API response
. by commenting/uncommenting $schema. Worth to note it could also be any other reason, like invalid Gemini API key.
Hi @.joshuafrank I have added the fix to promptfoo so please let me know if in the next release you have the same issue and I’ll fix! Thanks for sharing the issue
u
Yes. I can confirm that the issue is now resolved. Thank you @adelmuursepp__