Hi
@ian
Firstly, I want to say thank you for the Promptfoo framework, it has been invaluable in expanding my knowledge in LLMs and prompt engineering.
I’ve recently started to explore RAG and the benefit this can bring in increasing the accuracy that LLM’s respond with. If possible, would I be able to get some advice on how to configure my RAG pipeline with the Promptfoo framework?
For context...
I have a dataset in a csv file, which has the following format:
fruit,Nutritional-Value-1,Nutritional-Value-2,Nutritional-Value-3
Apple,95 calories,0 gram fat,1 gram protein
Banana,110 calories, 0 gram fat, 1 gram protein
I have followed the “RAG from Scratch” guide by Mistral AI; and the text embedding loaded into open-source vector database, FAISS.
I have an existing “tests.csv” which has nearly 50 tests in, which has following format:
fruit,__expected1,__expected2
Apple, model-graded-closedqa:0 gram fat, model-graded-closedqa:1 gram protein
For non-RAG LLM evaluations, I would pass the prompt:
“Tell me the nutritional value of one {{fruit}}”.
Now, sorry for the silly question here; but how do I evaluate my RAG pipeline? I’ve followed the guide in
https://promptfoo.dev/docs/guides/evaluate-rag/ but I just do not understand it.
How can I load the vector database into the LLM I wish to use (in this instance, mistral) and utilise the existing tests file?
Any guidance or clarification would be greatly appreciated. Thank you in advance for your help!