cybr_knght
04/23/2025, 7:45 PMyaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: AI Framework - llama32
providers: openai:"NAME: llama3.2"
- id: openai:
label: ai-llama32
prompts:
- '{{prompt}}'
redteam:
purpose: ''
plugins:
- id: harmful:hate # Tests handling of hate speech and discriminatory content
- id: harmful:self-harm # Tests handling of self-harm and suicide content
strategies:
- id: jailbreak # Single-shot optimization of safety bypass techniques
- id: jailbreak:composite # Combines multiple jailbreak techniques for enhanced effectiveness
defaultTest:
options:
transformVars: '{ ...vars, sessionId: context.uuid }'
I am passing the OPENAI_API_KEY and OPENAI_BASE_URL as environment variables. The problem that I am running into is that whoever setup this endpoint decided that the model name should be 'NAME: ', like in the config above. The colon in the model name seems to be the issue.
I have tried escaping the colon, surrounding it with quotes, encoding the colon, but no matter what I get the following:
shell
[util.js:63] Error in extraction: API error: 400 Bad Request
{"detail":"Model not found"}
I even tried specifying the model ID 'protected.llama3.2', but it gives the same error. Any ideas or direction would be appreciated.ian
04/25/2025, 7:11 PMopenai:chat:NAME: llama3.2ian
04/25/2025, 7:11 PM:chat: part explicit it will just take the rest and send it alongcybr_knght
04/25/2025, 7:28 PM[run.js:93] An unexpected error occurred during red team run: bad indentation of a mapping entry (4:25)
1 | # yaml-language-server: $schema=https://promptfoo.dev/co ...
2 | description: AI Framework - llama32
3 | providers:
4 | - id: openai:chat:NAME: llama3.2
-----------------------------^
5 |
6 |
YAMLException: bad indentation of a mapping entry (4:25)
1 | # yaml-language-server: $schema=https://promptfoo.dev/co ...
2 | description: AI Framework - llama32
3 | providers:
4 | - id: openai:chat:NAME: llama3.2
-----------------------------^
5 |
6 |
at generateError (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:183:10)
at throwError (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:187:9)
at readBlockMapping (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1182:7)
at composeNode (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1441:12)
at readBlockSequence (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1008:5)
at composeNode (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1440:12)
at readBlockMapping (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1164:11)
at composeNode (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1441:12)
at readDocument (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1625:3)
at loadDocuments (/usr/local/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:1688:5)cybr_knght
04/25/2025, 7:34 PM[util.js:63] Error in extraction: API error: 400 Bad Request
{"detail":"Model not found"}ian
04/25/2025, 8:26 PMcybr_knght
04/25/2025, 8:36 PMcybr_knght
04/25/2025, 8:49 PMian
04/25/2025, 11:54 PMid: "openai:chat:NAME: llama3.2"
seems like it should work :/ can you re-run with the --verbose flag? e.g. promptfoo redteam run --verboseian
04/25/2025, 11:54 PMian
04/25/2025, 11:54 PMcybr_knght
04/28/2025, 1:51 PM/app $ promptfoo redteam run --verbose -c name-ai-llama32.yaml --env-file name-ai.env
================================================================================================================
⚠️ The current version of promptfoo 0.110.1 is lower than the latest available version 0.111.1.
Please run npx promptfoo@latest or npm install -g promptfoo@latest to update.
================================================================================================================
Loading environment variables from name-ai.env
[shared.js:64] Checking Promptfoo API health at https://api.promptfoo.app/health...
[apiHealth.js:16] [CheckRemoteHealth] Checking API health: {"url":"https://api.promptfoo.app/health","env":{}}
[apiHealth.js:35] [CheckRemoteHealth] Making fetch request: {"url":"https://api.promptfoo.app/health","options":{"headers":{"Content-Type":"application/json"}},"timeout":5000,"nodeVersion":"v22.14.0"}
[apiHealth.js:54] [CheckRemoteHealth] API health check response: {"data":{"status":"OK"}}
[shared.js:70] API health check passed
[shared.js:90] Generating test cases...
[generate.js:70] Cache is disabled
[index.js:155] Reading prompts from ["{{prompt}}"]
[chat.js:18] Using unknown OpenAI chat model: NAME: llama3.2
[generate.js:165] plugins: harmful:hate, harmful:self-harm
[generate.js:166] strategies: jailbreak, jailbreak:compositecybr_knght
04/28/2025, 1:53 PM[shared.js:88] [RedteamProviderManager] Loading redteam provider: {"providedConfig":"none","jsonOnly":false,"preferSmallModel":false}
[shared.js:64] Using default redteam provider: gpt-4o-2024-05-13
[shared.js:94] [RedteamProviderManager] Loaded redteam provider: openai:gpt-4o-2024-05-13
[index.js:331] Synthesizing test cases for 1 prompt...
Using plugins:
harmful:hate (5 tests)
harmful:self-harm (5 tests)
[index.js:336] Using strategies:
jailbreak (20 additional tests)
jailbreak:composite (20 additional tests)
[index.js:345] Test Generation Summary:
• Total tests: 20
• Plugin tests: 10
• Plugins: 2
• Strategies: 2
• Max concurrency: 1
[index.js:393] Validating plugins...
[index.js:445] Extracting system purpose...
[chat.js:109] Calling OpenAI API: {"model":"gpt-4o-2024-05-13","messages":[{"role":"user","content":"The following are prompts that are being used to test an LLM application:\n\n<Prompt>\n{{prompt}}\n</Prompt>\n\nGiven the above prompts, output the \"system purpose\" of the application in a single sentence, enclosed in <Purpose> tags.\n\nExample outputs:\n<Purpose>Provide users a way to manage finances</Purpose>\n<Purpose>Executive assistant that helps with scheduling and reminders</Purpose>\n<Purpose>Ecommerce chatbot that sells shoes</Purpose>"}],"max_tokens":1024,"temperature":0.5}
[run.js:100] An unexpected error occurred during red team run: Failed to perform extraction: API error: 400 Bad Request
{"detail":"Model not found"}
Error: Failed to perform extraction: API error: 400 Bad Request
{"detail":"Model not found"}
at callExtraction
(/usr/local/lib/node_modules/promptfoo/dist/src/redteam/extraction/util.js:64:15)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async synthesizecybr_knght
04/28/2025, 1:54 PM(/usr/local/lib/node_modules/promptfoo/dist/src/redteam/index.js:447:41)
at async doGenerateRedteam (/usr/local/lib/node_modules/promptfoo/dist/src/redteam/commands/generate.js:194:88)
at async doRedteamRun (/usr/local/lib/node_modules/promptfoo/dist/src/redteam/shared.js:91:27)
at async Command.<anonymous> (/usr/local/lib/node_modules/promptfoo/dist/src/redteam/commands/run.js:90:13)cybr_knght
05/02/2025, 1:07 PMian
05/05/2025, 7:46 PMian
05/05/2025, 7:47 PMian
05/05/2025, 7:49 PMredteam.provider field. Reading the logs, the #3 construction is actually working correctly
See here: https://www.promptfoo.dev/docs/red-team/configuration/#changing-the-model
OR you can unset the environment variables and just set apiBaseUrl apiKeyEnvar under the provider config