https://promptfoo.dev/ logo
Setting up a common grader for LLM tests (differen...
# ❓|questions
e
Hi! I've been testing several RAG configurations using the provided context-relevance, context-faithfulness and answer-relevance commands. My goal is to evaluate the generation of 10 golden queries, so I can see which configurations perform better. While doing my tests, I've discovered a few issues with the process: - When you set a provider for the defaultTest, it's used both for generation and evaluation. - When small models (SLM) are used for evaluation, they tend to fail responding "No output". Maybe then can't properly follow the underlying prompt? - When setting a larger model using the --grader CLI tool, it complains when performing "Answer relevancy check", falling back to the SLM and skewing the results. (Sample error:
Provider ollama:chat:command-r:latest is not a valid embedding provider for 'answer relevancy check', falling back to default
) -Message
Using OpenAI default providers
is printed when in verbose mode, even when a provider is both set for text and embeddings. Given all these issues at hand, what is the best way to config a grader for RAG that is different from the model being used to generate the responses? Find in the next message my sample config file.
Sample config file:
Copy code
description: "RAG - Command-R test prompts"

prompts:
  - file://prompts/prompt_baseline.txt
  - file://prompts/prompt_xml.txt


providers:
  - id: ollama:chat:command-r:latest
    label: command-r
    config:
      temperature: 0
      stream: "false"
      system: "You are an expert journalist that created compeling stories about local events"
  
defaultTest:
  options:
    provider:
      text:
        id: ollama:chat:command-r:latest
      embedding:
        id: ollama:embeddings:nomic-embed-text:latest
  assert:
    - type: context-relevance
      threshold: 0.8
    - type: context-faithfulness
      threshold: 0.8
    - type: answer-relevance
      threshold: 0.8

tests:
  - vars:
      query: "What event happened a few years ago at Fablab Alicante?"
      context: file://golden_queries_documents/q1.txt
In this example, Command-R is being used both for generation and evaluation.
This is a sample config file to test Gemma2:2B as generator and Command-R as grader using Ollama as the provider.
Copy code
description: "Gemma generator - Command-R grader"

prompts:
  - file://prompts/prompt_baseline.txt
  - file://prompts/prompt_xml.txt


providers:
    - id: ollama:chat:gemma2:2b-instruct-q8_0
    label: gemma2_2b
    config:
      temperature: 0
      stream: "false"
      system: "You are an expert journalist that created compeling stories about local events"


defaultTest:
  options:
    provider:
      text:
        id: ollama:chat:gemma2:2b-instruct-q8_0
      embedding:
        id: ollama:embeddings:nomic-embed-text:latest
  assert:
    - type: context-relevance
      threshold: 0.8
    - type: context-faithfulness
      threshold: 0.8
    - type: answer-relevance
      threshold: 0.8

tests:
  - vars:
      query: "What event happened a few years ago at Fablab Alicante?"
      context: file://golden_queries_documents/q15.txt
I launch the evaluation with this command:
promptfoo eval -c 3_model_generation_gemma_grader_command_r_mini.yaml --verbose --no-cache
getting this response:
Copy code
Successes: 1
Failures: 1
Pass Rate: 50.00%
Token usage: Total 0, Prompt 0, Completion 0, Cached 0
When setting the grader via CLI using this command
promptfoo eval --grader ollama:chat:command-r:latest -c 3_model_generation_gemma_grader_command_r_mini.yaml --verbose --no-cache
I am getting an error in my response:
Copy code
TypeError: textProvider.callApi is not a function

TypeError: textProvider.callApi is not a function
    at matchesContextRelevance (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:504:37)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async runAssertion (/usr/local/lib/node_modules/promptfoo/dist/src/assertions.js:898:17)
    at async /usr/local/lib/node_modules/promptfoo/dist/src/assertions.js:1138:24
(Damm, I was not getting this error in previous versions (0.73 or similar) but I am getting it right now with 0.78.3. This error happens when specifing a grader in the CLI besides the default test model.
Along with that error, I am getting this message in the output too:
Copy code
Using OpenAI default providers
Using OpenAI default providers
Using OpenAI default providers
Provider ollama:chat:command-r:latest is not a valid embedding provider for 'answer relevancy check', falling back to default
Using OpenAI default providers
Note that I am not using any OpenAI default providers (checked in my OpenAI console to make sure I was not using any model)
According to the docs, the four ways to Override GPT-4 as LLM grader are: - Using --grader CLI option (doesn't work) - Passing it as a parameter for defaultTest.options (wouldn't not work as the same provider would be used for generation and grading - Passing it as a parameter for test.options - Passing it as a parameter for assertion provider Let's check those two last options
In my config file, I want to execute the same test with several multiple queries, so the assertions are set at defaultTest level. My first try was:
Copy code
description: "Gemma generator - Command-R grader"

prompts:
  - file://prompts/prompt_baseline.txt
  - file://prompts/prompt_xml.txt


providers:
  - id: ollama:chat:gemma2:2b-instruct-q8_0


defaultTest:
  options:
    provider:
      text:
        id: ollama:chat:gemma2:2b-instruct-q8_0
        label: gemma2_2b
        config:
          temperature: 0
          stream: "false"
          system: "You are an expert journalist that created compeling stories about local events"
      embedding:
        id: ollama:embeddings:nomic-embed-text:latest
  assert:
    - type: context-relevance
      threshold: 0.8
    - type: context-faithfulness
      threshold: 0.8
    - type: answer-relevance
      threshold: 0.8

tests:
  - description: Event question
    vars:
      query: "What event happened a few years ago at Fablab Alicante?"
      context: file://golden_queries_documents/q15.txt
    options:
      provider: ollama:chat:command-r:latest
But got the error:
Provider ollama:chat:command-r:latest is not a valid embedding provider for 'answer relevancy check', falling back to default
So, setting the provider for text at test level, without specifying it's a text provider, will block the use of the embeddings defined at default test level.
What happens if I define at test level a text provider?
Copy code
yaml
description: "Gemma generator - Command-R text grader"

prompts:
  - file://prompts/prompt_baseline.txt
  - file://prompts/prompt_xml.txt


providers:
  - id: ollama:chat:gemma2:2b-instruct-q8_0


defaultTest:
  options:
    provider:
      text:
        id: ollama:chat:gemma2:2b-instruct-q8_0
        label: gemma2_2b
        config:
          temperature: 0
          stream: "false"
          system: "You are an expert journalist that created compeling stories about local events"
      embedding:
        id: ollama:embeddings:nomic-embed-text:latest
  assert:
    - type: context-relevance
      threshold: 0.8
    - type: context-faithfulness
      threshold: 0.8
    - type: answer-relevance
      threshold: 0.8

tests:
  - description: Event question
    vars:
      query: "What event happened a few years ago at Fablab Alicante?"
      context: file://golden_queries_documents/q15.txt
    options:
      provider:
        text:
          id: ollama:chat:command-r:latest
Will it mask the text provider defined at defaultTest level? Nope, it will fail complaning that command-R is not a valid embeddings provider:
Copy code
Error: Invalid provider definition for output type 'embedding': {
  "text": {
    "id": "ollama:chat:command-r:latest"
  }
}

Error: Invalid provider definition for output type 'embedding': {
  "text": {
    "id": "ollama:chat:command-r:latest"
  }
}
    at getGradingProvider (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:82:19)
    at getAndCheckProvider (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:91:35)
    at matchesAnswerRelevance (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:396:37)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async runAssertion (/usr/local/lib/node_modules/promptfoo/dist/src/assertions.js:881:17)
    at async /usr/local/lib/node_modules/promptfoo/dist/src/assertions.js:1138:24
This setup works but will make me repeat the same options block per question:
Copy code
description: "Gemma generator - Command-R text grader plus embeddings"

prompts:
  - file://prompts/prompt_baseline.txt
  - file://prompts/prompt_xml.txt


providers:
  - id: ollama:chat:gemma2:2b-instruct-q8_0


defaultTest:
  options:
    provider:
      text:
        id: ollama:chat:gemma2:2b-instruct-q8_0
        label: gemma2_2b
        config:
          temperature: 0
          stream: "false"
          system: "You are an expert journalist that created compeling stories about local events"
      embedding:
        id: ollama:embeddings:nomic-embed-text:latest
  assert:
    - type: context-relevance
      threshold: 0.8
    - type: context-faithfulness
      threshold: 0.8
    - type: answer-relevance
      threshold: 0.8

tests:
  - description: Event question
    vars:
      query: "What event happened a few years ago at Fablab Alicante?"
      context: file://golden_queries_documents/q15.txt
    options:
      provider:
        text:
          id: ollama:chat:command-r:latest
        embedding:
          id: ollama:embeddings:nomic-embed-text:latest
Let's explore the final way to configure it at assetion level. In this particular case, I will configure them up as options under defaultTest.asserts:
Copy code
yaml
description: "Gemma generator - Command-R text grader plus embeddings - assets"

prompts:
  - file://prompts/prompt_baseline.txt
  - file://prompts/prompt_xml.txt

providers:
  - id: ollama:chat:gemma2:2b-instruct-q8_0

defaultTest:
  options:
    provider:
      text:
        id: ollama:chat:gemma2:2b-instruct-q8_0
        label: gemma2_2b
        config:
          temperature: 0
          stream: "false"
          system: "You are an expert journalist that created compeling stories about local events"
      embedding:
        id: ollama:embeddings:nomic-embed-text:latest
  assert:
    - type: context-relevance
      threshold: 0.8
      options:
      provider:
        text:
          id: ollama:chat:command-r:latest
        embedding:
          id: ollama:embeddings:nomic-embed-text:latest
    - type: context-faithfulness
      threshold: 0.8
      options:
      provider:
        text:
          id: ollama:chat:command-r:latest
        embedding:
          id: ollama:embeddings:nomic-embed-text:latest
    - type: answer-relevance
      threshold: 0.8
      options:
      provider:
        text:
          id: ollama:chat:command-r:latest
        embedding:
          id: ollama:embeddings:nomic-embed-text:latest
tests:
  - description: Event question
    vars:
      query: "What event happened a few years ago at Fablab Alicante?"
      context: file://golden_queries_documents/q15.txt
This final configuration looks quite nicer than the previous one for multiple questions 🙂
a
I'm seeing this issue with a very minimal config as well:
Copy code
yaml
description: 'minimal repo'

prompts:
  - 'test prompt {{input}}'

providers:
  - "openai:chat:gpt-4o"

defaultTest:
  assert:
    - type: llm-rubric
      value: 'Check if response is good'

tests:
  vars:
    - input: my test
Running:
npx promptfoo@latest eval --grader openai:gpt-3.5-turbo --no-cache
I get:
Copy code
TypeError: finalProvider.callApi is not a function

TypeError: finalProvider.callApi is not a function
    at matchesLlmRubric (/home/arseni/.npm/_npx/1754c693ca488390/node_modules/promptfoo/dist/src/matchers.js:249:38)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async runAssertion (/home/arseni/.npm/_npx/1754c693ca488390/node_modules/promptfoo/dist/src/assertions.js:846:17)
    at async /home/arseni/.npm/_npx/1754c693ca488390/node_modules/promptfoo/dist/src/assertions.js:1138:24
Also on a related note, I believe the docs for the "options > provider" are bit inaccurate. In the [docs](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/#overriding-the-llm-grader) it says:
Copy code
yaml
defaultTest:
options:
provider: openai:gpt-3.5-turbo
tests:
  - description: Evaluate output using LLM
    assert:
      - type: llm-rubric
        value: Is written in a professional tone
When the "provider" should be indented by 1 level to be parsed correctly. I'm thinking the "rubric prompt" text later on probably has the same issue but I haven't tested that
i
👀 thank you for persisting, elsatch
thanks for flagging - just fixed this
e
I think this could make an interesting guide on the docs!
a
Thanks for being on this so quickly! I'm really enjoying the tool so far
e
I am preparaing a question about the RAG retriever.py files as I find the current distinction between retriever and generator quite confusing but while I have the time to process all current examples, I have another meta-question: Would it make sense to have a Grader class that is a subclass from Provider, so it can be configured separately? It could make the configuration and logical separatoin much more clear and simple.
i
yeah, I think that would help substantially
one common footgun is people set their custom provider, but don't realize they need to [override the grader](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader) as well