https://promptfoo.dev/ logo
OpenAI Compatible API name problems
# ❓|questions
c
I am trying to run a redteam scan against an openAI compatible model running in a custom Open WebUI application. At this point I am running the plugins on the "minimal test" preset plugin configuration. Here is the configuration file:
Copy code
yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: AI Framework - llama32
providers: openai:"NAME: llama3.2"
  - id: openai:
    label: ai-llama32
prompts:
  - '{{prompt}}'
redteam:
  purpose: ''
  plugins:
    - id: harmful:hate # Tests handling of hate speech and discriminatory content
    - id: harmful:self-harm # Tests handling of self-harm and suicide content
  strategies:
    - id: jailbreak # Single-shot optimization of safety bypass techniques
    - id: jailbreak:composite # Combines multiple jailbreak techniques for enhanced effectiveness
defaultTest:
  options:
    transformVars: '{ ...vars, sessionId: context.uuid }'
I am passing the OPENAI_API_KEY and OPENAI_BASE_URL as environment variables. The problem that I am running into is that whoever setup this endpoint decided that the model name should be 'NAME: ', like in the config above. The colon in the model name seems to be the issue. I have tried escaping the colon, surrounding it with quotes, encoding the colon, but no matter what I get the following:
Copy code
shell
[util.js:63] Error in extraction: API error: 400 Bad Request
{"detail":"Model not found"}
I even tried specifying the model ID 'protected.llama3.2', but it gives the same error. Any ideas or direction would be appreciated.
i
can you try doing
openai:chat:NAME: llama3.2
I think if you make the
:chat:
part explicit it will just take the rest and send it along
c
No luck
Copy code
[run.js:93] An unexpected error occurred during red team run: bad indentation of a mapping entry (4:25)

 1 | # yaml-language-server: $schema=https://promptfoo.dev/co ...
 2 | description: AI Framework - llama32
 3 | providers:
 4 |   - id: openai:chat:NAME: llama3.2
-----------------------------^
 5 | 
 6 | 
YAMLException: bad indentation of a mapping entry (4:25)

 1 | # yaml-language-server: $schema=https://promptfoo.dev/co ...
 2 | description: AI Framework - llama32
 3 | providers:
 4 |   - id: openai:chat:NAME: llama3.2
-----------------------------^
 5 | 
 6 | 
    at generateError (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:183:10)
    at throwError (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:187:9)
    at readBlockMapping (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1182:7)
    at composeNode (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1441:12)
    at readBlockSequence (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1008:5)
    at composeNode (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1440:12)
    at readBlockMapping (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1164:11)
    at composeNode (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1441:12)
    at readDocument (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1625:3)
    at loadDocuments (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1688:5)
If I surround the provider with quotes "openai:NAME: llama3.2". it still says
Copy code
[util.js:63] Error in extraction: API error: 400 Bad Request
{"detail":"Model not found"}
i
Try "openai:chat:TAMU: llama 3.2" with quotes and the chat prefix
c
I tried it both ways "openai:chat:NAME: llama3.2" same error as above "Model not found"
The examples my team has given me use the model id 'protected.llama3.2' instead of the model name "NAME: llama3.2". Maybe I should open a GH issue to allow the openai provider to use either the model name or ID?
i
Copy code
id: "openai:chat:NAME: llama3.2"
seems like it should work :/ can you re-run with the
--verbose
flag? e.g.
promptfoo redteam run --verbose
otherwise yeah, probably need to make a change on our side
sorry for the back and forth
c
Okay, I have tried the following: 1) openai:chat:"NAME: llama3.2" -> result -> "An unexpected error occurred during red team run: bad indentation of a mapping entry (4:25)" 2) openai:chat:NAME: llama3.2 -> result -> "An unexpected error occurred during red team run: bad indentation of a mapping entry (4:25)" 3) "openai:chat:NAME: llama3.2" -> result -> "Error in extraction: API error: 400 Bad Request" Log - part 1:
Copy code
/app $ promptfoo redteam run --verbose -c name-ai-llama32.yaml --env-file name-ai.env 

================================================================================================================
⚠️ The current version of promptfoo 0.110.1 is lower than the latest available version 0.111.1.

Please run npx promptfoo@latest or npm install -g promptfoo@latest to update.
================================================================================================================

Loading environment variables from name-ai.env
[shared.js:64] Checking Promptfoo API health at https://api.promptfoo.app/health...
[apiHealth.js:16] [CheckRemoteHealth] Checking API health: {"url":"https://api.promptfoo.app/health","env":{}}
[apiHealth.js:35] [CheckRemoteHealth] Making fetch request: {"url":"https://api.promptfoo.app/health","options":{"headers":{"Content-Type":"application/json"}},"timeout":5000,"nodeVersion":"v22.14.0"}
[apiHealth.js:54] [CheckRemoteHealth] API health check response: {"data":{"status":"OK"}}
[shared.js:70] API health check passed
[shared.js:90] Generating test cases...
[generate.js:70] Cache is disabled
[index.js:155] Reading prompts from ["{{prompt}}"]
[chat.js:18] Using unknown OpenAI chat model: NAME: llama3.2
[generate.js:165] plugins: harmful:hate, harmful:self-harm
[generate.js:166] strategies: jailbreak, jailbreak:composite
Log part 2
Copy code
[shared.js:88] [RedteamProviderManager] Loading redteam provider: {"providedConfig":"none","jsonOnly":false,"preferSmallModel":false}
[shared.js:64] Using default redteam provider: gpt-4o-2024-05-13
[shared.js:94] [RedteamProviderManager] Loaded redteam provider: openai:gpt-4o-2024-05-13
[index.js:331] Synthesizing test cases for 1 prompt...
Using plugins:

harmful:hate (5  tests)
harmful:self-harm (5  tests)

[index.js:336] Using strategies:

jailbreak (20 additional tests)
jailbreak:composite (20 additional tests)

[index.js:345] Test Generation Summary:
• Total tests: 20
• Plugin tests: 10
• Plugins: 2
• Strategies: 2
• Max concurrency: 1

[index.js:393] Validating plugins...
[index.js:445] Extracting system purpose...
[chat.js:109] Calling OpenAI API: {"model":"gpt-4o-2024-05-13","messages":[{"role":"user","content":"The following are prompts that are being used to test an LLM application:\n\n<Prompt>\n{{prompt}}\n</Prompt>\n\nGiven the above prompts, output the \"system purpose\" of the application in a single sentence, enclosed in <Purpose> tags.\n\nExample outputs:\n<Purpose>Provide users a way to manage finances</Purpose>\n<Purpose>Executive assistant that helps with scheduling and reminders</Purpose>\n<Purpose>Ecommerce chatbot that sells shoes</Purpose>"}],"max_tokens":1024,"temperature":0.5}
[run.js:100] An unexpected error occurred during red team run: Failed to perform extraction: API error: 400 Bad Request
{"detail":"Model not found"}
Error: Failed to perform extraction: API error: 400 Bad Request
{"detail":"Model not found"}
    at callExtraction 
(/usr/local/lib/node_modules/promptfoo/dist/src/redteam/extraction/util.js:64:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async synthesize
Log part 3:
Copy code
(/usr/local/lib/node_modules/promptfoo/dist/src/redteam/index.js:447:41)
    at async doGenerateRedteam (/usr/local/lib/node_modules/promptfoo/dist/src/redteam/commands/generate.js:194:88)
    at async doRedteamRun (/usr/local/lib/node_modules/promptfoo/dist/src/redteam/shared.js:91:27)
    at async Command.<anonymous> (/usr/local/lib/node_modules/promptfoo/dist/src/redteam/commands/run.js:90:13)
What do you think? Should I create a GH issue or is this something on my side?
i
sorry for the delay here. #3 should work. I would recommend opening a github issue
as a workaround, you could also create a custom provider instead of using the OpenAI provider Javascript: https://www.promptfoo.dev/docs/providers/custom-api/ Python: https://www.promptfoo.dev/docs/providers/python/
OH you need to update the
redteam.provider
field. Reading the logs, the #3 construction is actually working correctly See here: https://www.promptfoo.dev/docs/red-team/configuration/#changing-the-model OR you can unset the environment variables and just set
apiBaseUrl
apiKeyEnvar
under the provider config