https://promptfoo.dev/ logo
Multiple __expected values in CSV input
# ❓|questions
v
Heya! Is it possible to have multiple
__expected
asserts with input vars coming from a csv similar to how you can have multiple asserts for YAML input? If not, is this something that may be supported in the future? Right now we are getting around it by creating multiple runs where the vars
__expected
changes depending on what we are trying to test.
for instance, currently we are running two
vars.csv
that look as follows:
Copy code
#vars1.csv
var1,var2,__expected
a,b,similar(0.8):....
and
Copy code
#vars2.csv
var1,var2,__expected
a,b,is_json
but it would be nice if we could have something similar to
Copy code
var1,var2,__expected,__expected
a,b,similar(0.8):...,is_json
i
Unfortunately not possible currently, but I like your suggested solution
v
awesome, thanks for the reply 🙂
i
I'll see if I can implement it in the next day or two (or something similar)
v
no worries, we're in the beginning stages of getting this all set up, so it isn't a super urgent thing!
for context, we are dynamically creating the input vars from a db of logged production runs, so this will make that testing a bit easier! thanks again!
i
@vitmont This PR adds support for any column that begins with
__expected
, so you can do
__expected1
,
__expected2
, ...,
__expectedN
for example. It will be included in the next release soon: https://github.com/promptfoo/promptfoo/pull/284
v
Appreciate the quick work! This works perfectly
9 Views