gustavisaksen
04/17/2024, 8:12 AMian
04/18/2024, 4:40 AMQuestion -> Subquestion and Subquestion -> Retrieval steps and test them separately. OR
- Write a custom provider (in python, js, or your preferred language) that runs the whole thing e2e, and have a bunch of test cases where each individual test case returns the result for a given subquestion
Both are imperfect approaches and not quite what you asked for.
I'm open to ideas on how we can update the configuration to support this.gustavisaksen
04/18/2024, 10:38 AMian
04/20/2024, 11:13 PMpromptfoo :: ragas
context-faithfulness :: Formerly RAGAS, but it appears they have removed this metric
context-relevance :: RAGAS Context Relevancy
factuality :: OpenAI evals model-graded factuality (not ragas)
answer-relevance :: RAGAS Answer Relevance
If there are RAGAS metrics that would be useful to you that aren't reflected here, LMK and we can add them pretty quickly!