https://promptfoo.dev/ logo
Running assert on each test item in a csv
# ❓|questions
j
I can run each item in a csv using
- file: input.csv
but im struggling to also run, I've tried a bunch of things, just looking for some direction ie:
Copy code
- label: 'input'
      # input: file://input.csv      
      threshold: 0.5
      file: file://input.csv      
      # vars:
        # goal: file://linput.csv     #
      assert:
       - type: javascript
         value: XXX
i can get it loading in as one big prompt, so the whole csv is one prompt, but not each line is a prompt
from what i can tell in the docs i can only do that from the same file using
__expected
?
so, i added ",file://assert.js as __expected in my csv, and everything fails with
Expected output XXXX  to equal "file://assert.js"
> file:// indicates an external file relative to your config. For example: file://custom_assertion.py or file://customAssertion.js
m
have a look at https://www.promptfoo.dev/docs/configuration/parameters/#tests-file the top key in your promptfooconfig.yaml should look something like
Copy code
tests: file://path/to/tests.csv
j
thats the closest i could get,
tests: file://file.csv
which has lines such as :
> input,__expectedinput string,file://customAssertion.js
but it is treating the __expectedInput as a string match, not looking at the customAssertion.js file
which then failed every line with
Expected output XXXX  to equal "file://customAssertion.js"
m
can you share your config and a csv with ~1/2 rows?
b
__expected should be something like ,fn:output.includes('cat')