vitmont
11/16/2023, 7:47 PM__expected asserts with input vars coming from a csv similar to how you can have multiple asserts for YAML input? If not, is this something that may be supported in the future? Right now we are getting around it by creating multiple runs where the vars __expected changes depending on what we are trying to test.vitmont
11/16/2023, 7:52 PMvars.csv that look as follows:
#vars1.csv
var1,var2,__expected
a,b,similar(0.8):....
and
#vars2.csv
var1,var2,__expected
a,b,is_json
but it would be nice if we could have something similar to
var1,var2,__expected,__expected
a,b,similar(0.8):...,is_jsonian
11/16/2023, 7:53 PMvitmont
11/16/2023, 7:55 PMian
11/16/2023, 7:55 PMvitmont
11/16/2023, 7:57 PMvitmont
11/16/2023, 7:57 PMian
11/17/2023, 1:27 AM__expected, so you can do __expected1, __expected2, ..., __expectedN for example. It will be included in the next release soon: https://github.com/promptfoo/promptfoo/pull/284vitmont
11/17/2023, 2:43 AM