https://promptfoo.dev/ logo
promptfoo continues to use openai even if I config...
# ❓|questions
u
Copy code
targets:
  - id: https
    label:
    config:
      url: "https://my-target.tld/eval"
      method: "POST"
      headers:
        Content-Type: "application/json"
        Authorization: "Bearer xxxxx"

      body: {"text":"{{prompt}}"}
redteam:
  purpose: "something."
  numTests: 5
  plugins:
    - hijacking  # Tests for unauthorized resource usage and purpose deviation
    - pii:api-db  # Tests for PII exposure via API/database access
    - pii:direct  # Tests for direct PII exposure vulnerabilities
    - pii:session  # Tests for PII exposure in session data
    - pii:social  # Tests for PII exposure via social engineering
    - politics  # Tests handling of political content and bias

  # Attack methods for applying adversarial inputs
  strategies:
    - basic 
    - jailbreak # Single-shot optimization of safety bypass techniques
    - jailbreak:composite # Combines multiple jailbreak techniques for enhanced effectiveness
  provider: deepseek:deepseek-chat
I set the DEEPSEEK_API_KEY var and run promptfoo redteam eval (I already generated prompts with promptfoo redteam generate), but it continue to try openai key, getting this error in the console `Target error: OpenAI API key is not set. Set the OPENAI_API_KEY environment variable or add
apiKey
to the provider config.. Full response: {"output":"","error":"OpenAI API key is not set. Set the OPENAI_API_KEY environment variable or add
apiKey
to the provider config.","to}` hoe can I force it to use the model confiured in the promptfooconfig.yaml?
i
hmm I don't see anything obviously wrong with this setup @michaelmichaelmichael what do you think? looks like they're overriding
redteam.provider
with
deepseek-chat
, but the target still looks for the OpenAI key for some reason
m
It may be the llm-rubric that needs a separate override for grading instead of the generation provider? Let me try to reproduce
u
this happens when I run eval manually with the cli, for example I ran "promptfoo redteam eval" it still try to get the openai key and fails, if I run the eval from the web UI with the same conf it works
but let's assume that I just ran "promptfoo redteam generate" and want to use the same tests with 2 different endpoint, how can I pass the redteam.yaml file to the conf? the problem is that I can't see it in the UI and if I try to run it with cli I always get this openai fail
if the problem is llm-rubric, in my case I don't need it, I mean the output that my endpoint generates has already a verdict, I just need to check it with a condition if result == 'true' or something like that. Is there a way to not run llm-rubric?
m
Please try pulling the latest version. We just shipped a fix for the OPENAI_API_KEY issue on the moderation API for redteams. We automatically added a moderation assertion to harmful outputs. Re. llm-rubric - you can override the grading provider like this: https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/#overriding-the-llm-grader