Hello promptfoo team, We are considering using promptfoo's remote Grading and Attack generation features. When using these features, we understand that a POST request is sent to the https://api.promptfoo.app/api/v1/task endpoint. The request body of this call may contain sensitive and confidential information from our product under development, such as prompts, context, and test cases. From a security and confidentiality standpoint, we need to clarify your data handling policies. Could you please let us know: Is the content of the request body (prompts, variables, etc.) sent to this endpoint persistently stored on promptfoo's servers (e.g., in databases or log files)? If it is stored, for how long is this data retained? Is our understanding correct that the data is deleted immediately after the task (grading/attack generation) is executed? We need to ensure our sensitive data is handled appropriately. We would greatly appreciate it if you could provide details on your data retention policy. Thank you.

Good question. When you use remote generation, request data gets logged to Google Cloud Logging for 90 days, then deleted. API keys, passwords, and other sentive information are scrubbed, but parts of prompts and test cases may show up in the logs. Nothing goes into a Promptfoo database. If you need everything local, set

PROMPTFOO_DISABLE_REMOTE_GENERATION=true

. For enterprise customers, we can work out custom data handling and DPAs (including shipping you an on prem instance so your data never leaves your system). If you do not want promptfoo to handle grading you can set a --grader or specify it in your redteam config. I'll update the privacy policy to make thsi more clear. Thanks for asking.

Thank you for your quick and clear response. I have one follow-up question to ensure I fully understand the policy, as we do want to use the remote features safely. Your previous answer mentioned that "When you use remote generation" data is logged for 90 days. To be perfectly clear: Does this logging policy (where prompts/test cases may be logged) also apply to remote Grading? Or, does the 90-day logging policy only apply to remote Attack Generation (via api/v1/task), and remote Grading (which also uses api/v1/task) has a different policy (e.g., data is not logged or is deleted immediately)? Thank you again for clarifying.

The logging policy applies to everything where we have a logger.info, logger.warn, logger.error, statement. the prompts are truncated. A real recent example (note how litle useful information is logged) is:

{
  "message": "[Task Route] Received request: judge",
  "task": "judge",
  "request": {
    "jsonOnly": true,
    "prompt": "[{\"role\":\"system\",\"content\":\"You are a helpful red teaming assistant...\"}]",
    "step": "judge",
    "task": "judge"
  }
}

If you don't want us to process the outputs of your llm, I recommend setting the grader. If you need greater privacy, please send us an email.

Thank you for providing the log example for the judge task. This is extremely helpful. I have one final, specific question regarding the llm-rubric task, as this is the grader we are planning to use. When we send a POST request for llm-rubric grading, our request body looks like the following. As you can see, the sensitive information is not in a "prompt" field, but rather in the "output" and "vars" fields. JSON

{
  "task": "llm-rubric",
  "email": "user.email@example.com",
  "rubric": "Our grading rubric details...",
  "output": "The actual full response generated by the LLM...",
  "vars": {
    "user_input": "The actual user input example...",
    "context": "Our application context..."
  }
}

Your previous answer confirmed that "prompts are truncated" in the logs. My question is: Does this truncation policy also apply to the other sensitive fields in this llm-rubric request? Specifically: Is the "output" field (which contains the LLM's full response) truncated in the logs? Are the contents of the "vars" object (like user_input and context) also truncated? This is the most critical point for us, as the output and vars fields will contain the confidential data we need to protect. Thank you again for your transparency and help in understanding the exact logging behavior.