So I want to create list of text examples, consisting of (question, documents from knowledge base, answer) and then use it to test separately each part of my pipeline, so
- retriever tests will use only question as input and documents for assertions
- llm tests will use question and documents as input, and answer for assertions
- e2e tests will use question as input, and documents+answer for assertions.
Now, I can't do it by using test.csv - __expected* columns are different for each of the above. I'm thinking about putting scenarios into scenarios.yaml, and then load it for each (retriever, llm, e2e) config file. Any suggestions?