eyad_llama
01/15/2024, 7:45 AMeyad_llama
01/15/2024, 8:12 AMian
01/16/2024, 7:44 PMk4fu
11/12/2024, 10:53 PMpromptfoo eval invocation and see the aggregated results.
My understanding, after digging through the docs, is that even if the prompts are in their own config files, the assertions will get applied to all the prompts. So, what I'd want as 4 tests, 2 for each prompt, really gets executed as 8 tests because the tool will create a matrix of all prompts and var/assertions?
Is that correct? If so, is there a workaround or plans to support coupling prompts and assertions together rather than matrixing them?k4fu
11/12/2024, 10:56 PMpromptfoo calls and then inspect the results of each?k4fu
11/13/2024, 3:31 PMk4fu
11/13/2024, 6:13 PMk4fu
11/13/2024, 9:16 PMmessages var and then doing the {{ messages | dump }} for the prompt. So, each individual test case contains everything it needs inside the test vars