https://promptfoo.dev/ logo
Idea: Dummy pandas provider
# ❓|questions
e
Today I was working on RAG evals and found several use cases where I'd rather load a baseline pipeline from file than having promptfoo running all evals again. (I know about the cache option). So I just realized that some of these scenarios could be solved with a simple dummy provider that would load the data from a csv file using pandas. Then in the config file you'd call the dummy provider to populate results and trigger the eval. One sample case, would be incorporating human annotated responses as a fake provider for comparison. Do you think this would be feasible / useful?
i
Might be an imperfect solution but there is an
echo
provider. What people do is they create a csv with all the data, load those as test cases, then run it through echo
e
Any link to the
echo
provider? I have searched the docs but I couldn't find it
i
hmm it may not be documented, but the implementation is literally just this:
Copy code
ts
  } else if (providerPath === 'echo') {
    ret = {
      id: () => 'echo',
      callApi: async (input: string) => ({ output: input }),
    };
baked into promptfoo's provider handler
e
This week I am testing several evaluation frameworks and it's quite interesting to notice the differences between those, in terms of input, output handling, etc.
i
What's your takeaway? I know the config based approach is a bit different, some people love it but others hate it 😬
e
Main takeaways (so far, I am just in the middle of evaluation so... final results might vary) My particular use case is multilingual RAG evaluation (over private company data), so for other uses cases the results might be different. Going back to basics. Lots of people mention RAGAS and ARES a RAG evaluation frameworks but IMHO: - RAGAS is not relevant as most new evaluation frameworks include RAGAS as another metric and their solution is just English prompts finetuned for GPT-4 and doesn't work that well for non english documents, non GPT models (aka local models). They created a fined tuned LLM-as-judge model based on Qwen-1.5 but it's only English AFAIK.
- ARES: very interesting approach but the code to implement it doesn't work at all so... Then I moved to other eval frameworks, in particular DeepEval, Promptfoo and Giskard. Factors to pin point those: langfuse and llamaindex integration, open source. Thoughts on promptfoo: - The good: very nice to do A/B testing, quite modular to be tailored to different metrics, good integrations. Once you get the RAG part tuned, you can red-team your model without too many changes. The team is quite responsive and active pushing fixes around. - The bad: once you are out of the 2-3 built-in use cases, you have to do all the other 12-13 manually. Out of the RAG documentation examples in the docs, only the docs describing the scripts exist. No other use cases are developed in full (yet they are prosposed as potential usages for the tool). I am not sure this tool will scale when dealing with multiple hyperparameters (aka whole RAG system performance)
Thoughts on Giskard: - The good: the whole RAGET evaluation feature looks really great, advising you on how to improve the over RAG. They document the support for local models as well as open ai models. Python based. Several cool features for QandA generation like topic extraction, answer quality per topic, etc. Datasets are managed using pandas - The bad: RAGET seems like an add-on over a ML evaluation framework. RAGET means you get holistic evaluations of the system, but doing A/B testing requires generating the whole systems from scratch. Several interesting cases are undocumented or can't be done simply (like embeddings A vs B evaluation). Their discord feels quite dead to me.
I am still figuring out DeepEval but: - The good: they offer the most metrics out of the box. The python library looks simple to use but the options to configure it... not. Seems tailored to holistic but also enables A/b testing. - The bad: everything in their docs and software seems business model focused. This means tons of ways to save your data on their systems, as a given default, plus the nice beautiful output UI being only available on their cloud too.So, I feel dodging funnels to their cloud service all the time. I will keep you updated, but maybe in another thread 🙂
Side question: Is there any way to do some kind of multiple hyperparameter testing using promptfoo? Right now, I assume everything is A/B testing as in model A vs B outputs, embeddings A vs B, but not a combination of those.
i
wow @elsatch, I somehow missed this thread but I really appreciate the thoughtful comparisons. gives us something to think about as we keep building the project. by hyperparameter testing, do you mean exploring all the knobs of model level (e.g. temperature) and retrieval (e.g. scoring, # docs)? right now that would mean having to set up a provider for each combination
sorry if this is a dumb question, but what are the "other use cases" that you want to see more examples of?
e
I think this link explains the topic better and more succintly that I could https://docs.confident-ai.com/docs/guides-optimizing-hyperparameters TLDR; What combination of RAG elements provide better results?
I think promptfoo is great for A/B testing: Is llama 3.1 better than Mistral 7b for RAG?
But once you start thinking like: What is the prompt and model that perform better... What is the parser, model and prompt... It's super hard to keep track of those
And a prompt can perform nice for a model or version (like the claude tags) and yet perform badly for another
There are several cases described high level, yet there is no code to support them. One example: vector store comparison. The docs: it's just using the retrieve_pinecone.py, retrieve_milvus.py and all these providers. The repo: retrieve_pinecone.py. No file found.
I need to check my notes to pinpoint specific use cases but for RAG: Things that are developed: - Model A vs B for generation (with Context Relevance, Answer Relevance, Faithfulnes, etc) - Prompt A vs B for generation Note: all this things are great and work easy if you are using static retrieval from files. Things that are described: - Vector store A vs B for retrieval - Dynamic retrieval for generation (there is the call_api example, but focused mostly on the Langchain example) Things that I think that they are not done (or are not trivial to setup): - Embeddings A vs B for retrieval storage, retrieval and generation - Chunk size A vs B - Top similarity K A vs B - Parser A vs B (might be done using context variable as static, but requieres you to create those manually) - Index structure A vs B - Rerankers A vs B (this is custom Python code probably) In general, these can be done if you adhere to custom python code and move the variables to the provider logic instead of the yaml params. But then yoou can loose track of the parameters if those are outside the yaml file.
@ian My suggestion would be to implement the missing examples using an existing framework like LlamaIndex and/or Haystack that have support to change elements and load them easily.
i
makes a lot of sense. I appreciate the feedback 🙂
definitely need to go deeper on RAG reference implementations and examples
this is on my todo list!
e
Hey Ian, I have just seen the documentation for echo. It looks great! https://www.promptfoo.dev/docs/providers/echo/
Going back to the some people love the config based approach, some other people don't... I feel like promptfoo like system could be great as a lightweight data quality tool. Working in a similar scheme, you could assert data quality metrics, based on differents tests... I will keep you updated!
i
if there are any metrics that are missing but would be useful to add, I'm interested in your suggestions
e
Main challenge is that these data quality metrics are a bit tangent to the prompt space. For example, one simple data quality metric to implement would be "is_valid_datetime" where you check if the output is formatted according to the ISO 8601 standard