https://promptfoo.dev/ logo
Single test database for different pipeline aspect...
# ❓|questions
w
So I want to create list of text examples, consisting of (question, documents from knowledge base, answer) and then use it to test separately each part of my pipeline, so - retriever tests will use only question as input and documents for assertions - llm tests will use question and documents as input, and answer for assertions - e2e tests will use question as input, and documents+answer for assertions. Now, I can't do it by using test.csv - __expected* columns are different for each of the above. I'm thinking about putting scenarios into scenarios.yaml, and then load it for each (retriever, llm, e2e) config file. Any suggestions?
i
hmm.. I'm not totally sure. Generally I just split out configs into small and focused test suites, similar to how I would organize unit tests. Then I run them separately. I don't actually use scenarios, but I know many others do