https://promptfoo.dev/ logo
Redteam Grader
# ❓|questions
u
Hello All, I wanted to test grading my redteam prompt responses locally and therefore I set the provider to an example Ollama Model, namely llama3.2. (please see right half of the attached screenshot, which shows my yaml). However, when I run redteam action, I still see that "llm_rubric" going to api.promptfoo.app/api/v1/task endpoint to assess the outcome. Please see the logs on the left half of the screen. The logs mention overriding of grader component to ollama:llama3.2, but still uses promptfoo remote endpoint. Am I configuring my yaml incorrectly? Thanks a lot. https://cdn.discordapp.com/attachments/1428369155555590244/1428369156109107200/Screenshot.png?ex=68f23ff0&is=68f0ee70&hm=ea2c15440b1483828504e1a8c4afb1a813cdde83835722ec1219745f14f4e1f4&
m
Hi Umut! You need to add the provider field inside the redteam block, not at the top level. Update your config like this: redteam: provider: ollama:chat:llama3.2 # Add this line purpose: '' plugins: - id: excessive-agency - id: harmful:hate strategies: - id: basic numTests: 1 maxConcurrency: 4 The grading provider resolution checks: 1. defaultTest.provider (if set) 2. Falls back to redteam.provider ✅ Alternatively, you can force local-only grading with: PROMPTFOO_DISABLE_REMOTE_GENERATION=true promptfoo eval -c your-config.yaml There's an official example showing this pattern: examples/redteam-provider-override/promptfooconfig.yaml
u
Thanks a lot, Michael. I will try immediately.
Hello @michaelmichaelmichael , it worked only after I set PROMPTFOO_DISABLE_REMOTE_GENERATION=true. Without this environment setting, llm-rubric requests were sent to the remote endpoint even if I had the local LLM under redteam. I am also a little bit confused with the terms, for example I see "grader" setting in the command line options but in the yaml file it's called "provider". Or, the environment variable is called PROMPTFOO_DISABLE_REMOTE_GENERATION, but I am interested in blocking the "evaluations" of request-response pairs. By setting this environment variable, the promp generations from api.promptfoo.app also stopped. What I mean is keeping the remote prompt generation but disabling the "llm-rubric" tasks to promptfoo endpoint (i.e. an environment variable like PROMPTFOO_DISABLE_PROMPTFOO_LLM_RUBRIC_ENDPOINT=true). I am just trying to make suggestions, please ignore them if they are invalid. I love promptfoo overall. Thanks for creating such a great product.
m
You are absolutely correct and thank you for the feedback. I am fixing the override and trying to make the docs more clear in https://github.com/promptfoo/promptfoo/pull/5959 please let me know if you have any additional feedback
u
+