Hi, I am looking at https://www.promptfoo.dev/docs/red-team/plugins/ and I've configured a redteam config that uses the pii plugin - which according to the docs doesn't require promptfoo's servers for inference. I've setup my provider as vertex:gemini-2.5-flash but when I run promptfoo redteam generate with -v I see calls to promptfoo's APIs for remote inference even with

PROMPTFOO_DISABLE_REMOTE_GENERATION=true

and

PROMPTFOO_SELF_HOSTED=1

Is using my own LLM for redteam generation just not supported at all?

Thanks for the detailed question - this is a common point of confusion. Yes, you can use your own LLM for redteam generation with plugins like

pii

, but there are a couple of things to sort out. 1:

PROMPTFOO_SELF_HOSTED

doesn't control remote generation This env var is for template security in Docker deployments - it doesn't affect remote API calls. You can drop it. 2: The "remote calls" you're seeing are probably telemetry, not test generation With

PROMPTFOO_DISABLE_REMOTE_GENERATION=true

, the PII plugin will use your local provider. But you'll still see calls to promptfoo servers for telemetry and health checks unless you also set:

bash
PROMPTFOO_DISABLE_TELEMETRY=true

To run locally with your own LLM

bash
PROMPTFOO_DISABLE_REMOTE_GENERATION=true \
PROMPTFOO_DISABLE_TELEMETRY=true \
promptfoo redteam generate -c your-config.yaml

Make sure your config specifies your provider for generation:

yaml
redteam:
  provider: vertex:gemini-2.5-flash
  plugins:
    - pii

The

pii

plugins, along with

contracts

,

hallucination

,

excessive-agency

,

imitation

,

overreliance

,

politics

,

prompt-extraction

,

rbac

,

shell-injection

,

sql-injection

, and the dataset-based plugins (

beavertails

,

cyberseceval

,

donotanswer

, etc.) all support local generation. Plugins marked with 🌐 in the docs (like

harmful:*

,

bias:*

, and several others) require remote generation because they use Promptfoo's specialized models. If you want to generate all plugins with your own inference, contact us about an on-prem deployment.

I ran it again with the two env vars above and it does as you say, so thank you for that. I'm worried about the "full featuredness" of running it this way however. For instance, I see this in the logs:

[logger.js:324] Extracting goal for 6 tests from pii:api-db...
[logger.js:324] Remote generation disabled, skipping goal extraction

Does that mean I'm getting a lesser quality run for the

pii

plugin than if I was using on-prem w/ my LLM or going direct to Promptfoo's servers? I'm trying to make an assessment on if we really need enterprise or not - I want to use the redteaming module and I was aware of the 🌐 restrictions.