https://promptfoo.dev/ logo
Evaluation of a RAG chain with sub questions
# ❓|questions
g
Hi, I am developing a QnA chatbot. I am trying to set up a chain, capable of splitting a question into subquestions, and for each subquestion retrieve 5 relevant documents. (see figure) I would like to be able to calculate a number of model graded metrics for each sub question, which i cant figure out how to do using promptfoo. 1. I need to be able to pass the sub questions forwards to the next chain, and dynamically change the vars field in the next chain. 2. Based on the dynamically updated vars field, i need to dynamically update my context (which is currently supported). Is type of setup supported in promptfoo? https://cdn.discordapp.com/attachments/1230068436860932116/1230068437015859200/image.png?ex=6631f980&is=661f8480&hm=29fbf32db85e438e8d243f80e52c10d3ea8f27a2d618904b01da00ca90da1853& https://cdn.discordapp.com/attachments/1230068436860932116/1230068437275901992/image.png?ex=6631f981&is=661f8481&hm=5dd94ffdabbad94f4a3897a695c296481c2a0af4387cae31526ad75cd4eb0abd&
i
Hi @gustavisaksen, right now the best way to test this sort of setup is: - Break down the
Question -> Subquestion
and
Subquestion -> Retrieval
steps and test them separately. OR - Write a custom provider (in python, js, or your preferred language) that runs the whole thing e2e, and have a bunch of test cases where each individual test case returns the result for a given subquestion Both are imperfect approaches and not quite what you asked for. I'm open to ideas on how we can update the configuration to support this.
g
Hi @User, thanks for your response. I think solution 1 would be the easier way of doing this with the current implementation. I have written a custom provider and context retriever, which i am currently using. However, since the provider is the one generating the subquestions, i would not be able to pass a specific context to the given subquestion. In my case i would need for the provider and the context retrieval to be able to handle lists or dicitionaries of question : context pairs. Alternatively, if promptfoo could provide an evaluate() function that takes in a question, context, answer and ground truth which is then appended to the output table in the frontend. RAGAS supports the evaluate function, which i have decided to use for now. On RAGAS: My understanding is that the modelgraded evaluations in promptfoo uses several of the RAGAS metrics, however it seems that the naming is not correctly mapped between the two docs. Could you point me to a place where i can see the mappings of: context-faithfulness, context-relevance, factuality and answer-relevance?
i
Hey @gustavisaksen, here's a mapping for the model graded metrics you mentioned:
Copy code
promptfoo :: ragas

context-faithfulness :: Formerly RAGAS, but it appears they have removed this metric
context-relevance :: RAGAS Context Relevancy
factuality :: OpenAI evals model-graded factuality (not ragas)
answer-relevance :: RAGAS Answer Relevance
If there are RAGAS metrics that would be useful to you that aren't reflected here, LMK and we can add them pretty quickly!
2 Views