https://promptfoo.dev/ logo
Multiple test files for same prompt
# ❓|questions
f
I'm trying to define a single
prompfooconfig.yaml
with a shared system_prompt variable, something like:
Copy code
providers:
  - file://gpt-3.5-turbo.yaml
  - file://gpt-4-preview.yaml

defaultTest:
  vars:
    system_prompt: file://system_prompt.txt
then have tests define in separate files, like
Copy code
prompts: [search_results.json]
tests:
  - vars:
      text: ...
I thought I could run this with
promptfoo eval -t tests/*.yaml
But it gives me this error:
You must provide at least 1 prompt
What's the recommended way to have this kind of separation? Thanks!
i
The test files are expected to be lists of test cases, not including prompts:
Copy code
yaml
- vars:
    text: test A
- vars:
    text: test B
You'd have to put the prompts at the top of the main config, or use multiple configs (
promptfoo eval -c promptfooconfig1.yaml promptfooconfig2.yaml
). Let me know how that works for you!
f
The problem is that when I use
-c
with multiple prompts then each test is run with all prompts, and that's not what I expected. What I want to do is something like, have a
promptfooconfig.yaml
like this:
Copy code
yaml
providers:
  - file://config/gpt-3.5-turbo.yaml
  - file://config/gpt-4-preview.yaml

defaultTest:
  vars:
    system_prompt: file://config/system_prompt.txt
and then, multiple test files like this:
Copy code
yaml
prompts: [search_results.json]
tests:
  - vars:
      text: Find my orders
What I'm trying to accomplish is to simulate open AI conversations and function calls, my pompts look like this:
Copy code
json
[
    {
        "role": "system",
        "content": "{{system_prompt}}"
    },
    {
        "role": "user",
        "content": "{{text}}"
    }
]
I don't mind if I have to duplicate the settings in each config file, but when I tried
eval -c config1.yaml config2.yaml
they got merged and each test was executed with both prompts (from
config1
and
config2
) but the prompts from
config1
only make sense with then tests for
config1
, does that make sense to you?
If that's not supported that's fine, I can run them individually (that's what I'm actually doing), but would be nice to have a way to run like
eval -c promptfooconfig.yaml -t tests/*.yaml
But the prompts in my case are defined in each test file, but It's expected to be on the config.
20 Views