https://promptfoo.dev/ logo
Running multiple sets of tests, do not give the ex...
# ❓|questions
d
I have two tests
promptfooconfig_1.yaml
and
promptfooconfig_2.yaml
. They are different tests. If I ran them individually they pass, but if i run them together with:
promptfoo eval -c ./*
they fail, because it seems to me they mix together the vars or the prompts of each other. However I would like to run everything in one single command without mixing them. Is there a shorthand to run all the tests in a folder and subdirectory and have a single summary, without this side effect? Thank you.
m
We support patterns that approximate this behavior. Check out this example where we add labels to prompts to filter which model runs them: https://github.com/promptfoo/promptfoo/tree/main/examples/prompt-labels If this doesn't work for you I recommend running them separately. Unfortunately, the UI is not really designed to display completely separate datasets. If you tell me what you need from the summary I can help brainstorm through munging it with you
d
Hi @michaelmichaelmichael, thank you for your support, it is appreciated. Let me explain in a bit more detail what is my use case and maybe we can try to figure out something. I have an application I am trying to set up a CICD prompt testing step. The application has a router, has a RAG and a chatbot. Each component is semi-independent from each other. Furthermore the chatbot for example, can be in a conversation. I have created a few yaml config files trying to test the various components in both a single and a conversation. For example:
Copy code
description: "Router prompt testing: Single message"

prompts:
  - file://../prompt_one_message.jsonl

providers:
  - id: "openai:gpt-4o"
    config:
      temperature: 0.0
# Set up the conversation history
defaultTest:
  vars:
    system_prompt: file://sys_prompt.txt
    user_prompt: file://user_prompt.txt
tests:
  - vars:
      query: 'What is the value of sales?'
    assert:
      ...some asserts...

  - vars:
      query: 'what is the value of revenue?'
    assert:
        ...some asserts...
I then have other configs like this one:
Copy code
description: "chat-final-response-call: Two turns conversation"

prompts:
  - file://../prompt_two_messages.jsonl

providers:
  - id: "python:provider.py"
    config:
      temperature: 0.1
      model: gpt-4o

# Set up the conversation history
defaultTest:
  vars:
    system_prompt: file://chat_system_prompt.txt
    user_prompt_template: chat_prompt.txt

tests:

  - vars:
      first_prompt: what is the value of net income?
      assistant_response: file://mock_messages/assistant_response_net_income.txt
      _enriched_context_dict:  file://mock_messages/company_context.json
      .... some flags...
      query: why is it increasing?
    assert:
      ...some asserts...
If I run these test separately they work, but if I run them togheter they don't. The reason I want to have a single summary is because I want to run these steps in a CICD pipeline and have a single overview of the success/failures. If I run these tests separately how can I do that? feel free to ask for more info.
m
Thank you for the context! Are you running the promptfoo action or did you write your own ci/cd flow? If your own, I recommend running them one after another and just reporting the output from both. You can fail if the exit code from either one returns non-0. If via the promptfoo action, setup a step for each. If easier to resolve on a call, feel free to grab time on my calendar https://cal.com/michael-dangelo/15min
d
Hi @User , thank you so much for your support. I think I get it. Yes I do have my own CICD flow in azuredevops. Just a pipeline. I use the html and the json output to publish the tests results. I can create a single script that executes all the tests and that will work, what it is a bit awkward maybe is that I have to later process all the output files with another script to merge and collect all the results. I guess it should not be too big deal but I was hoping to be able to generate in a easy way a single output file. If you have a better idea please let me know. Thank you.