Hi all,
I am working for a cybersecurity service provider and we would like to use Promptfoo to test LLM applications of our customers. Data privacy is of major importance to us, and we therefore don't want to send any data or requests of any sort to PromptFoo's cloud services. In practice, this means that adversarial input generation, response evaluation and grading of attacks should all happen in our systems, and that all telemetry should be disabled. Looking at the documentation (
https://www.promptfoo.dev/docs/red-team/configuration/#how-attacks-are-generated), we have several questions regarding the correct configuration to use:
Will setting the PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION env var to true prevent adversarial input generation requests to be sent to promptFoo's API, while allowing us to use our own remote LLM deployed in our cloud environment? Or should we specify our own attacker model provider in the config file, while leaving PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION to its default value, false?
Additionally, I understand that it is possible to override the default grader by specifying a custom one in the config file:
https://www.promptfoo.dev/docs/red-team/troubleshooting/grading-results/#overriding-the-grader.
Will making those two configuration changes (specifying a custom attacker model provider, and a custom grader) be enough to ensure that no data (including telemetry of usage data) is ever sent to promptFoo's services? If not, what additional configuration is needed to achieve this?
Thanks