https://promptfoo.dev/ logo
The rubricPrompt property works only for the llm-r...
# ❓|questions
l
Hello! I need to override the default prompt for the answer-relevance, context-relevance, and factuality metrics because the LLM RAG I am using in my project interacts in Brazilian Portuguese. I believe that this language difference is affecting the evaluation results. Therefore, I think that overriding the metric prompts with versions in PT-BR would improve the evaluation outcomes. However, the documentation is not clear about whether the rubricPrompt property can be used with any model-grade metric. Could someone clarify this?
m
Hi @luizfelipe42 you can read through the assertion implementations here: https://github.com/promptfoo/promptfoo/blob/main/src/assertions.ts#L1014 and find many of the model grading prompts here https://github.com/promptfoo/promptfoo/blob/main/src/prompts/external/ragas.ts#L1 TLDR is that most support overriding rubricPrompt as a string but you should be careful because you may change the definition of the metric. I recommend just using llm-rubric as your metric for every test and overriding that if you go down the path of translating the ragas prompts
2 Views