Hi, I have a generated test case that is using the promptfoo:redteam:iterative jailbreak strategy. About a month ago, I set this up and when I’d run the eval for it, I would use --grader vertex:gemini-2.5-flash on the command line and it worked fine. Now I’m noticing that promptfoo tries to use gpt-5-2025-08-07 even when I set the grader on the command line. However, if in my redteam block I put redteam.provider: vertex:gemini-2.5-flash then it uses gemini. Any ideas what I might be doing wrong here? Did the behaviour change recently? Thanks

This PR will solve it: https://github.com/promptfoo/promptfoo/pull/7100 it will be in the release today

Awesome. Do you know which version it’ll be introduced in?

the release that just went out! 0.120.15

Hey so on v0.120.16 it appears to try loading Gemini but still uses OpenAI Logs: [logger-BHK-IvoX.js:774] [RedteamProviderManager] Loading redteam provider { "providedConfig": "none", "jsonOnly": true, "preferSmallModel": false } [logger-BHK-IvoX.js:764] Using default redteam provider: gpt-5-2025-08-07 [logger-BHK-IvoX.js:764] [RedteamProviderManager] Loaded redteam provider: openai:gpt-5-2025-08-07 [logger-BHK-IvoX.js:764] Using redteam provider: [Google Vertex AI Provider gemini-2.5-flash] [logger-BHK-IvoX.js:764] [RedteamProviderManager] Using grading provider from defaultTest: vertex:gemini-2.5-flash [logger-BHK-IvoX.js:764] [Iterative] Starting iteration 1/4 [logger-BHK-IvoX.js:774] Provider call failed during eval { "providerId": "python:../../providers/", "providerLabel": "", "promptIdx": 0, "testIdx": 0, "pluginId": "ascii-smuggling", "strategyId": "jailbreak", "error": {} } [logger-BHK-IvoX.js:764] Results: 0 passed, 0 failed, ✗ 1 error (0%) [logger-BHK-IvoX.js:764] Duration: 2s (concurrency: 4) Error message: Error: API key is not set. Set the OPENAI_API_KEY environment variable or add

apiKey

to the provider config.

Hey @dulax , sorry for the delay! You were right, PR #7100 only partially fixed the issue. It made --grader work for the grading provider, but the redteam provider was still defaulting to OpenAI. A follow-up fix was merged (https://github.com/promptfoo/promptfoo/pull/7236) which makes --grader control both providers. Can you upgrade to v0.120.18 or later and try again?