https://promptfoo.dev/ logo
Setting up the evaluation using the RAG schema/str...
# ❓|questions
f
Hello, I'd like to use the RAG that I already have in place in a project to test it in its current state but above all to be able to play with the different parameters (prompt, template, embeddings, vector store...) and compare the results. The aim would be to really have the same context as the RAG I've set up to get usable results. I have a fairly complex RAG with precise extraction methods depending on the type of document etc...and I'm wondering what's the best way to do it in my case ? Thanks in advance for your help!
m
@Fav read through and try running this example: https://github.com/promptfoo/promptfoo/blob/main/examples/rag-full/promptfooconfig.with-asserts.yaml where we play with the configuration of a RAG. We can also compare prompts there. If you are changing things like the embeddings and vector store you can define multiple [retrieve.py](https://github.com/promptfoo/promptfoo/blob/main/examples/rag-full/retrieve.py) files side by side after ingestion. I am happy to answer more specific questions after you get started.
f
Thanks a lot, I'm currently using this very example. The problem I'm having is that I can't use the complex prompt generated in retrieve.py instead of the '{{question}}' that is defined in the example. I can't figure out how to call it in the prompt section of the config file. I had to modify the retrieve.py file in order to use llama3 locally and to use other free libraries instead of openai, so maybe my modifications are the problem.
m
What you're doing is fine. You can keep the prompt in retrieve.py no problem. Set up two providers (retrieve.py, retrieve2.py etc) and add them to the providers array if you want to test different configurations
f
The problem is that I don't know what to set in the Prompt section of the config file to use the prompt generated in retrieve.py. In the screenshots I'm still using the '{{question}}' prompt but I would like to use the 'final prompt' generated in retrieve.py Because I'm assuming that just with the '{{question}}' prompt it takes the questions of the tests and that's it ?
m
ah. You can set it to anything if you'd like (pass-through, 1, etc), retrieve.py has access to all of the vars so you can ignore the prompt section of the promptfooconfig if you want and construct the prompt in your provider (you just need one item in prompts). Check out the context arg of call_api https://www.promptfoo.dev/docs/providers/python/#python-script
f
Thanks a lot for your help!
e
@User are you using LiteLLM to expose the Ollama server to Promptfoo?
f
Sorry I didn't saw your message, I'm using promptfoo in VS Code and I'm not using LiteLLM