https://promptfoo.dev/ logo
Trying to wrap my head about retrieve.py for RAG e...
# ❓|questions
e
Last week I shared my experience setting up the common grader to eval RAG models. I have been mapping this week the A/B tests that could be performed to evaluate RAG: different embedding models, parsers, rerankers, etc. There are tons of possibilities but most of them involve creating a custom provider (I am using LlamaIndex for that). One of the most confusing aspects of the whole process is the terminology. Reading the docs, all that is nedded is a retrieve.py script that will retrieve the docs...and also do the generation. Most RAG eval frameworks differenciate between the retrieve and the generation steps, but here the retrieve script is doing the whole enchilada. In particular, the retrieve script should: - Use a embeddings model to convert the query to numerical values - Query the vector datastore - Retrieve top_k fragments - Trigger the generation of the responses passing the fragments as context, along with a RAG prompt - Passing the results along with the fragments back to promptfoo Main issue I find with the approach is that the retrieve.py script ain't just used to get data from vector store, but performing all the other tasks using hyperparameters that won't be stored at all at the config file. Embedding model, number of fragments, prompt... everything is configured elsewhere but in the config file. Is there any way to pass the config options back to promptfoo so that they are registered too? Does anyone else find the terms a bit confusing for RAG evals?
i
Hmmm, are you suggesting that we actually split the retrieve vs generate abstraction on promptfoo's side? Or are you just saying that the docs should be rewritten to make these two steps distinct 😄
e
That is the retrieve.py script on the example RAG full example. This is the content:
Copy code
python
import os

from langchain.prompts import ChatPromptTemplate
from langchain.schema import HumanMessage
from langchain_community.vectorstores import Chroma
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
CHROMA_PATH = "db"

# Initialize embeddings and load the Chroma database
embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY)
db_chroma = Chroma(persist_directory=CHROMA_PATH, embedding_function=embeddings)

# Prompt template for generating answers
PROMPT_TEMPLATE = """
Answer the question based only on the following context:
{context}
Answer the question based on the above context: {question}.
Provide a detailed answer.
Don't justify your answers.
Don't give information not mentioned in the CONTEXT INFORMATION.
Do not say "according to the context" or "mentioned in the context" or similar.
"""


def call_api(prompt, options, context):
    k = options.get("config", {}).get("topK", 5)
    docs_chroma = db_chroma.similarity_search_with_score(prompt, k=k)
    context_text = "\n\n".join([doc.page_content for doc, _score in docs_chroma])

    # Generate prompt using the template
    prompt_template = ChatPromptTemplate.from_template(PROMPT_TEMPLATE)
    final_prompt = prompt_template.format(context=context_text, question=prompt)

    # Fetch from OpenAI API
    chat = ChatOpenAI(
        model_name="gpt-4o-mini", temperature=0, openai_api_key=OPENAI_API_KEY
    )
    message = HumanMessage(content=final_prompt)
    response = chat([message])

    result = {
        "output": response.content,
    }

    return result
That retrieve.py script does: Retrieval: - Defining the embeddings model for the retrieval - Set the vector database - Creating a prompt template - Perform retrieval using similarity score and topK 5 Generation: - Calling the OpenAI endpoint to generate the response Promptfoo wraping: - Setting the context and the response to the ProviderResponse that is expected by promptfoo So several key things to track when evaluating RAG are not in promptfooconfig.yaml anymore. They are defined in the external file. It would be great to be able to collect that data from the retrieve.py script and record it along topK in this particular example.
Checking the docs, I see that you use retrieve_and_generate.py, yet in the full example it's only called retrieve.py: https://www.promptfoo.dev/docs/guides/evaluate-rag/#evaluating-end-to-end-performance
BTW, I am mapping promptfoo eval features to the fifteen control points presented in this paper: https://arxiv.org/abs/2407.07858v1