https://promptfoo.dev/ logo
It looks like the open source tool uses Promptfoo'...
# ❓|questions
l
Are you all also collecting the results of the evals to identify which prompts/probes are the most effective?
i
we don't collect that information, although in the long run I'd like to allow users to share that data to improve red team effectiveness we have a few plugins that are server-side only because they require our custom models or in some cases, openai models (which not everyone can access)
l
Thanks for the clarification. What are the models used in a typical open source red teaming eval?
Also given the costs associated with hosting such models, will you all be offering that support in the long-run?
i
by default, the redteam uses: - GPT-4o whenever possible (this can be overridden to other providers using the
redteam.provider
option, although we find that GPT produces the best results) - Promptfoo models for specific harmful use cases (such as hate speech, self-harm) Right now we're covering the cost of Promptfoo models, but there are limits to the number of redteams we can support per user for free. If your org is going to run a lot of evals, recommend contacting us for commercial terms If you want to remain on free only, just omit the harmful plugins (or replace them your own [intents](https://www.promptfoo.dev/docs/red-team/plugins/intent/)) and it will behave like any other open-source red teaming tool.