dadatm
09/30/2024, 3:18 PMpromptfooconfig_1.yaml and promptfooconfig_2.yaml. They are different tests. If I ran them individually they pass, but if i run them together with: promptfoo eval -c ./* they fail, because it seems to me they mix together the vars or the prompts of each other. However I would like to run everything in one single command without mixing them. Is there a shorthand to run all the tests in a folder and subdirectory and have a single summary, without this side effect? Thank you.michaelmichaelmichael
09/30/2024, 4:03 PMdadatm
10/01/2024, 9:09 AMdescription: "Router prompt testing: Single message"
prompts:
- file://../prompt_one_message.jsonl
providers:
- id: "openai:gpt-4o"
config:
temperature: 0.0
# Set up the conversation history
defaultTest:
vars:
system_prompt: file://sys_prompt.txt
user_prompt: file://user_prompt.txt
tests:
- vars:
query: 'What is the value of sales?'
assert:
...some asserts...
- vars:
query: 'what is the value of revenue?'
assert:
...some asserts...
I then have other configs like this one:
description: "chat-final-response-call: Two turns conversation"
prompts:
- file://../prompt_two_messages.jsonl
providers:
- id: "python:provider.py"
config:
temperature: 0.1
model: gpt-4o
# Set up the conversation history
defaultTest:
vars:
system_prompt: file://chat_system_prompt.txt
user_prompt_template: chat_prompt.txt
tests:
- vars:
first_prompt: what is the value of net income?
assistant_response: file://mock_messages/assistant_response_net_income.txt
_enriched_context_dict: file://mock_messages/company_context.json
.... some flags...
query: why is it increasing?
assert:
...some asserts...dadatm
10/01/2024, 9:09 AMmichaelmichaelmichael
10/01/2024, 2:11 PMdadatm
10/01/2024, 9:52 PM