https://promptfoo.dev/ logo
Running multiple tests in one config file
# ❓|questions
e
Hey guys, In my system I have 4 different prompts that do different things. I want to create a test suit that tests all 4 prompts on a group of test cases that I have created for each test. I am struggling to figure out how do I have 4 different configs: Each prompt has a provider config, and let's say 10 tests (with assertions) associated with it. I want to run an eval that runs each prompt with it's provider config and its own test data . At the moment if I just list 4 prompts with all 40 test cases it runs each prompt on all 40 cases, when it should only be on 10 tests.
Even when I create mutliple config files, with each of them having a prompt, provider config and a number of cases, I still see that a prompt from one file is tested against cases from the other file
i
k
Hi @ian, sorry for hijacking this question, but I think I'm going to run into a similar situation as the OP. For context, let me share an example of what I'm trying to accomplish... ideally, the final state would be a test suite of prompt evals we can manually trigger. For the most basic example of this suite, say I have two prompts: - Write me a {{medium}} about {{topic}} - Tell me a fact about {{topic}} And I have two test cases for each... For the "write me" prompt: - Write me a short story about the ocean - Write me a haiku about a squirrel For the "tell me" prompt: - Tell me a fact about California - Tell me a fact about New York Ideally, I'd want to be able to run those 2 prompts and 2 tests each as a total of 4 tests using one
promptfoo eval
invocation and see the aggregated results. My understanding, after digging through the docs, is that even if the prompts are in their own config files, the assertions will get applied to all the prompts. So, what I'd want as 4 tests, 2 for each prompt, really gets executed as 8 tests because the tool will create a matrix of all prompts and var/assertions? Is that correct? If so, is there a workaround or plans to support coupling prompts and assertions together rather than matrixing them?
I think this [thread](https://discordapp.com/channels/1153767083381903389/1290331819329458290) is also similar, and it looks like that OP is having to make multiple
promptfoo
calls and then inspect the results of each?
Hmm, would using the default prompt per test var be a workaround for coupling one prompt to one set of assertions?
I don't think using the prompt passthrough works with multishot inputs
What I've ended up doing is building out the
messages
var and then doing the
{{ messages | dump }}
for the prompt. So, each individual test case contains everything it needs inside the test vars