https://promptfoo.dev/ logo
red teaming giving error
# ❓|questions
v
i was trying to use red teaming but it is giving error as below promptfoo generate redteam -w --no-cache Cache is disabled. Synthesizing test cases for 2 prompts... Plugins: excessive-agency, hallucination, harmful, hijacking, jailbreak, overreliance, pii, prompt-injection, harmful:violent-crime, harmful:non-violent-crime, harmful:sex-crime, harmful:child-exploitation, harmful:specialized-advice, harmful:privacy, harmful:intellectual-property, harmful:indiscriminate-weapons, harmful:hate, harmful:self-harm, harmful:sexual-content, harmful:cybercrime, harmful:chemical-biological-weapons, harmful:illegal-drugs, harmful:copyright-violations, harmful:misinformation-disinformation, harmful:harassment-bullying, harmful:illegal-activities, harmful:graphic-content, harmful:unsafe-practices, harmful:radicalization, harmful:profanity, harmful:insults █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 3% | ETA: 1s | 3/100C:\node_modules\promptfoo\node_modules\tiny-invariant\dist\tiny-invariant.cjs.js:14 throw new Error(value); ^ Error: Invariant failed: Expected purpose to be a string at invariant (C:\node_modules\promptfoo\node_modules\tiny-invariant\dist\tiny-invariant.cjs.js:14:11) at getPurpose (C:\node_modules\promptfoo\dist\src\redteam\index.js:137:34) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async synthesize (C:node_modules\promptfoo\dist\src\redteam\index.js:77:41) at async Command. (C:\node_modules\promptfoo\dist\src\main.js:407:30) Node.js v18.17.0
i
I'll look into this, as a workaround can you add an argument
--purpose ...
that describes the purpose of your system in 1 sentence For example,
--purpose 'Customer support bot for Acme, a company that sells XYZ'
v
with --purpose also same error coming....
i
What does your prompt config look like?
oh, other question is do you have
OPENAI_API_KEY
set?
m
Hi @vikash_45108 we just released a new version of promptfoo which adds the ability to set custom providers in redteam test generation (such as
--provider ollama:llama2-uncensored
) and it adds better debug logs for when purpose is not a string. Please give it another try and let us know if you're stuck. If you continue to be stuck I am happy to hop on a call with you at any point tomorrow.
v
I am using GROQ for doing this below is the config file: description: "My eval" prompts: - "Write a tweet about {{topic}}" - "Write a concise, funny tweet about {{topic}}" providers: - id: openai:chat:mixtral-8x7b-32768 config: apiBaseUrl: https://api.groq.com/openai/v1 apiKeyEnvar: GROQ_API_KEY defaultTest: options: provider: id: openai:chat:mixtral-8x7b-32768 config: apiBaseUrl: https://api.groq.com/openai/v1 apiKeyEnvar: GROQ_API_KEY tests: - vars: topic: bananas I am trying with latest realease of promptfoo, but still it is giving tghe same error. Thanks vikash kumar prasad
i
have you overridden the redteam provider?
v
so, basically you are saying if i have replaced promptfooconfig.yaml file another promptfooconfig.yaml file? yes, i replaced with my own where provider is GROQ. but i think this should not make any difference, right?
promptfoo generate redteam -w --no-cache --purpose 'Customer support bot for Acme, a company that sells XYZ' Cache is disabled. Synthesizing test cases for 2 prompts... Plugins: contracts, excessive-agency, hallucination, harmful, hijacking, jailbreak, overreliance, pii, politics, prompt-injection, harmful:violent-crime, harmful:non-violent-crime, harmful:sex-crime, harmful:child-exploitation, harmful:specialized-advice, harmful:privacy, harmful:intellectual-property, harmful:indiscriminate-weapons, harmful:hate, harmful:self-harm, harmful:sexual-content, harmful:cybercrime, harmful:chemical-biological-weapons, harmful:illegal-drugs, harmful:copyright-violations, harmful:misinformation-disinformation, harmful:harassment-bullying, harmful:illegal-activities, harmful:graphic-content, harmful:unsafe-practices, harmful:radicalization, harmful:profanity, harmful:insults ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 8% | ETA: 672s | 8/100 C:\users\vikas\AppData\Roaming\npm\node_modules\promptfoo\node_modules\tiny-invariant\dist\tiny-invariant.cjs.js:14 throw new Error(value); ^ Error: Invariant failed: Expected generatedPrompts to be a string at invariant (C:\Users\vikas\AppData\Roaming\npm\node_modules\promptfoo\node_modules\tiny-invariant\dist\tiny-invariant.cjs.js:14:11) at Object.getContractTests [as contracts] (C:\Users\vikas\AppData\Roaming\npm\node_modules\promptfoo\dist\src\redteam\getUnintendedContractTests.js:42:34) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async synthesize (C:\Users\vikas\AppData\Roaming\npm\node_modules\promptfoo\dist\src\redteam\index.js:113:33) at async Command. (C:\Users\vikas\AppData\Roaming\npm\node_modules\promptfoo\dist\src\main.js:436:30) Node.js v18.17.0 this is how we are running