fabioxgn
11/30/2023, 6:29 PMprompfooconfig.yaml with a shared system_prompt variable, something like:
providers:
- file://gpt-3.5-turbo.yaml
- file://gpt-4-preview.yaml
defaultTest:
vars:
system_prompt: file://system_prompt.txt
then have tests define in separate files, like
prompts: [search_results.json]
tests:
- vars:
text: ...
I thought I could run this with promptfoo eval -t tests/*.yaml
But it gives me this error: You must provide at least 1 prompt
What's the recommended way to have this kind of separation?
Thanks!ian
12/01/2023, 3:40 PMyaml
- vars:
text: test A
- vars:
text: test B
You'd have to put the prompts at the top of the main config, or use multiple configs (promptfoo eval -c promptfooconfig1.yaml promptfooconfig2.yaml). Let me know how that works for you!fabioxgn
12/01/2023, 4:31 PM-c with multiple prompts then each test is run with all prompts, and that's not what I expected.
What I want to do is something like, have a promptfooconfig.yaml like this:
yaml
providers:
- file://config/gpt-3.5-turbo.yaml
- file://config/gpt-4-preview.yaml
defaultTest:
vars:
system_prompt: file://config/system_prompt.txt
and then, multiple test files like this:
yaml
prompts: [search_results.json]
tests:
- vars:
text: Find my orders
What I'm trying to accomplish is to simulate open AI conversations and function calls, my pompts look like this:
json
[
{
"role": "system",
"content": "{{system_prompt}}"
},
{
"role": "user",
"content": "{{text}}"
}
]
I don't mind if I have to duplicate the settings in each config file, but when I tried eval -c config1.yaml config2.yaml they got merged and each test was executed with both prompts (from config1 and config2) but the prompts from config1 only make sense with then tests for config1, does that make sense to you?fabioxgn
12/01/2023, 4:34 PMeval -c promptfooconfig.yaml -t tests/*.yaml
But the prompts in my case are defined in each test file, but It's expected to be on the config.