yuriy.ward
10/15/2024, 11:43 AMquestion and __expected. The goal is to send the question to an assistant API and compare the response with the __expected output, which contains the ideal response. However, I'm facing issues when trying to define metrics for the entire file. I tried providing - file: and - assert:, but it fails. While defining metrics per row is possible, it's not convenient, and the factuality tests still fail when the assistant's response is a superset of the expected result.
I would like a way to define metrics for the entire file and allow for flexible comparisons where the assistant's response can include more details than the __expected output but still pass the test if the factual content is correct. This would help when the assistant provides additional context that doesn't invalidate the factuality of the response.
I have considered defining metrics per row, but this approach doesn't seem to work as expected for factuality tests. Even when the assistant's response is a superset of the expected result, the test fails. Ideally, I'd prefer a simpler way to handle this at the file level.
Example YAML configuration I'm using:
yaml
description: "Factuality evaluation of Assistant"
prompts:
- "{{question}}"
providers:
- openai:assistant:asst_id-XYZ
defaultTest:
options:
provider: openai:gpt-4o-mini
tests: file://short-testset.csv
Example CSV that fails:
csv
question,__expected,__metric
what GPT stands for?,"GPT stands for ""Generative Pre-training Transformer""",factuality
Ideal CSV:
csv
question,__expected
what GPT stands for?,"GPT stands for ""Generative Pre-training Transformer"""
Ideal YAML:
yaml
tests:
- file: file://short-testset.csv
assert:
- type: factuality
I'm looking for guidance on how to define the metrics and handle the superset comparison for factuality testing. Thanks in advance 🙏