https://promptfoo.dev/ logo
How to run factuality tests on a CSV file?
# ❓|questions
y
I'm trying to run factuality tests on a CSV file, where I have two columns:
question
and
__expected
. The goal is to send the
question
to an assistant API and compare the response with the
__expected
output, which contains the ideal response. However, I'm facing issues when trying to define metrics for the entire file. I tried providing
- file:
and
- assert:
, but it fails. While defining metrics per row is possible, it's not convenient, and the factuality tests still fail when the assistant's response is a superset of the expected result. I would like a way to define metrics for the entire file and allow for flexible comparisons where the assistant's response can include more details than the
__expected
output but still pass the test if the factual content is correct. This would help when the assistant provides additional context that doesn't invalidate the factuality of the response. I have considered defining metrics per row, but this approach doesn't seem to work as expected for factuality tests. Even when the assistant's response is a superset of the expected result, the test fails. Ideally, I'd prefer a simpler way to handle this at the file level. Example YAML configuration I'm using:
Copy code
yaml
description: "Factuality evaluation of Assistant"

prompts:
  - "{{question}}"

providers:
  - openai:assistant:asst_id-XYZ

defaultTest:
  options:
    provider: openai:gpt-4o-mini

tests: file://short-testset.csv
Example CSV that fails:
Copy code
csv
question,__expected,__metric
what GPT stands for?,"GPT stands for ""Generative Pre-training Transformer""",factuality
Ideal CSV:
Copy code
csv
question,__expected
what GPT stands for?,"GPT stands for ""Generative Pre-training Transformer"""
Ideal YAML:
Copy code
yaml
tests:
  - file: file://short-testset.csv
    assert:
      - type: factuality
I'm looking for guidance on how to define the metrics and handle the superset comparison for factuality testing. Thanks in advance 🙏