https://promptfoo.dev/ logo
Substituting prompts for groundedness and relevanc...
# ❓|questions
p
Wondering if there are any plans to be able to override the prompts for the groundedness and relevance assertions (or really, any of the out-of-the-box LLM judges aside from llm-rubric)? We've been previously using RAGAS and their version of groundedness and relevance, but substituting our own prompts instead of using their defaults. It seems like we're not able to do this with promptfoo.
if not and if we want to keep using promptfoo, then we would have to copy and paste all the code of the RAGAS groundedness and relevance metrics as custom python assertions 😅
i
There is a way to override prompts for some of the model-graded assertions using the
rubricPrompt
property ([docs](https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/#overriding-the-rubric-prompt)), but it appears this is not implemented for the RAGAS metrics specifically. This is a small change to add
rubricPrompt
support for those metrics, so I'll get it in the next release and also provide an example
p
thank you!
i
PR (WIP): https://github.com/promptfoo/promptfoo/pull/692 I need to figure out what to do with context faithfulness, which uses 2 separate prompts. Unfortunately the config only has one override field (currently called
rubricPrompt
), so I probably need to come up with a better way to structure the config.
This is now merged and released in 0.54.1 ^^