https://promptfoo.dev/ logo
Redteam generates only partial set of inputs, resu...
# ❓|questions
g
Hi, promptfoo newbie here. We are looking at redteam to evaluate our LLM. Trying to generate 5 tests per topic but for many of the harmful plugins its only making 1 of 5, those 4 missing ones cause failures when the eval runs... https://cdn.discordapp.com/attachments/1298680225839976589/1298680226146025565/message.txt?ex=671a71b6&is=67192036&hm=d7774a9ba7abca7882949d05f2dc77b4746dfad728720ed1b7513e270298be52&
i
👀 can you share your config? Are you overriding the redteam provider?
g
In trying to get to a minimally reproducible public example I can get generation to work if my purpose string is shorter. I'd used part of our actual prompt as the purpose string. Is there guidance on what should be in here for best results? Failing case: purpose: "Your role is to provide technical information to a professional network engineer performing network troubleshooting working in the telecommunications space on wireline and wireless networks. You are the professional expert of REDACTED COMPANY Inc. You must provide technically accurate and clear explanations of complex concepts. Tailor responses to the specific context of the question, ensuring that the information provided is directly relevant and applicable. Use correct and specific technical terminology in all responses to ensure clarity and precision. Where appropriate, include examples or analogies to help illustrate technical concepts and make them more understandable. Ensure that responses address all parts of the question comprehensively, leaving no aspect unanswered. " Working case: purpose: "Your role is to provide technical information to a professional network engineer performing network troubleshooting working in the telecommunications space on wireline and wireless networks."
other than purpose and the target script other settings to reproduce are as follows, setting provider to GPT-4o also gives the same results. redteam: numTests: 5 plugins: - harmful:unsafe-practices
i
Ahh interesting
Will look into this