https://promptfoo.dev/ logo
Difficult time trying to understand how to set a c...
# ❓|questions
u
I'm having a difficult time setting a custom provider for the redteam and defaultTest without the eval defaulting back to OpenAI. Is there an end to end example that anyone knows about showing how to set up the yaml file & script files for each spot where you can set up a custom a custom provider.
Copy code
description: My first red team
prompts:
  - file://./prompt.json
providers:
  - id: python:./main_provider_script.py
redteam:
  provider: 
    id: 'python:./redteam_provider_script.py'
  plugins:
    - competitors
    - contracts
    - harmful:chemical-biological-weapons
  strategies:
    - id: base64
    - id: crescendo
  purpose: >-
    this is the llm application I'm testing
  entities: &ref_0
    - Leo Smith
defaultTest:
  metadata:
    purpose: >-
      this is the llm application I'm testing
    entities: *ref_0
  options:
    provider:
      text: 
        id: python:./moderation_provider_script.py
      embedding:
        id: python:./moderation_provider_script.py
i
do you run into errors during generation, or during the eval itself?
We just released fixes for this in 0.81.2 and you should be able to run the full process end-to-end without an OpenAI API key. Here's an example [configuration](https://github.com/promptfoo/promptfoo/blob/main/examples/redteam-provider-override/promptfooconfig.yaml), as well as a pointer to the related [docs](https://www.promptfoo.dev/docs/red-team/configuration/#providers). Note that redteam generation is tested with OpenAI, so quality of attacks may vary depending on which model you're using. You are welcome to use the OpenAI key I provided. Let me know if you have any questions!
u
Hi Ian, I appreciate the response. 1: This was generated using Open-AI then difficulty changing providers came in during eval. Ideally I would like to use a custom provider for generation, and all steps of eval. 2: I'll take a look at the change logs and try reimplementing the solution. Also I haven't seen it, but you're providing a OpenAI API key for users? I'll stick to using mine, but I respect that.
i
oops, got my wires crossed and misremembered a different conversation in which I provided an OpenAI key
the fixes we landed and the example I showed should give you a totally OpenAI-free experience!
let me know if that doesn't work out
u
haha makes sense! will do Ian. I'll report back my experience! thanks
i
u
Hi Ian, just a quick update: I set up a custom red team provider. Other than some errors being thrown on some tests due to invalid json format being returned from the LLM..it's working fairly well. However, one issue I am seeing is the LLM Grader Provider specified under the
defaultTest->options->provider
doesn't appear to be being called. Instead the RedTeam provider appears to be doing the grading. Is this intended behaviour? Extra Info ----------- attempted the following configs:
promptfoo eval --verbose --grader python:./moderation_provider_script.py -c redteam.yaml
Copy code
defaultTest:
  metadata:
    purpose: >-
      LLM Application purpose
    entities: *ref_0
  options:
    provider:
      text: 
        id: python:./moderation_provider_script.py
      embedding:
        id: python:./moderation_provider_script.py
Copy code
defaultTest:
  metadata:
    purpose: >-
      LLM Application purpose
    entities: *ref_0
  options:
    provider: python:./moderation_provider_script.py
i
Ah I see what you're saying
I'll take a look
u
Appreciate your time Ian. I’m happy to provide assistance if need be.
i
u
Thanks Ian, I’ll give it a shot and report back.
i
It's not in a release yet (assuming you're not building from source) but I will include it in a release today!
u
just reporting back, the fix to provide your own grader works great
2 Views