elsatch
08/21/2024, 2:59 PMProvider ollama:chat:command-r:latest is not a valid embedding provider for 'answer relevancy check', falling back to default)
-Message Using OpenAI default providers is printed when in verbose mode, even when a provider is both set for text and embeddings.
Given all these issues at hand, what is the best way to config a grader for RAG that is different from the model being used to generate the responses?
Find in the next message my sample config file.elsatch
08/21/2024, 3:02 PMdescription: "RAG - Command-R test prompts"
prompts:
- file://prompts/prompt_baseline.txt
- file://prompts/prompt_xml.txt
providers:
- id: ollama:chat:command-r:latest
label: command-r
config:
temperature: 0
stream: "false"
system: "You are an expert journalist that created compeling stories about local events"
defaultTest:
options:
provider:
text:
id: ollama:chat:command-r:latest
embedding:
id: ollama:embeddings:nomic-embed-text:latest
assert:
- type: context-relevance
threshold: 0.8
- type: context-faithfulness
threshold: 0.8
- type: answer-relevance
threshold: 0.8
tests:
- vars:
query: "What event happened a few years ago at Fablab Alicante?"
context: file://golden_queries_documents/q1.txt
In this example, Command-R is being used both for generation and evaluation.elsatch
08/21/2024, 3:10 PMdescription: "Gemma generator - Command-R grader"
prompts:
- file://prompts/prompt_baseline.txt
- file://prompts/prompt_xml.txt
providers:
- id: ollama:chat:gemma2:2b-instruct-q8_0
label: gemma2_2b
config:
temperature: 0
stream: "false"
system: "You are an expert journalist that created compeling stories about local events"
defaultTest:
options:
provider:
text:
id: ollama:chat:gemma2:2b-instruct-q8_0
embedding:
id: ollama:embeddings:nomic-embed-text:latest
assert:
- type: context-relevance
threshold: 0.8
- type: context-faithfulness
threshold: 0.8
- type: answer-relevance
threshold: 0.8
tests:
- vars:
query: "What event happened a few years ago at Fablab Alicante?"
context: file://golden_queries_documents/q15.txtelsatch
08/21/2024, 3:22 PMpromptfoo eval -c 3_model_generation_gemma_grader_command_r_mini.yaml --verbose --no-cache getting this response:
Successes: 1
Failures: 1
Pass Rate: 50.00%
Token usage: Total 0, Prompt 0, Completion 0, Cached 0elsatch
08/21/2024, 3:30 PMpromptfoo eval --grader ollama:chat:command-r:latest -c 3_model_generation_gemma_grader_command_r_mini.yaml --verbose --no-cache I am getting an error in my response:
TypeError: textProvider.callApi is not a function
TypeError: textProvider.callApi is not a function
at matchesContextRelevance (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:504:37)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async runAssertion (/usr/local/lib/node_modules/promptfoo/dist/src/assertions.js:898:17)
at async /usr/local/lib/node_modules/promptfoo/dist/src/assertions.js:1138:24elsatch
08/21/2024, 3:31 PMelsatch
08/21/2024, 3:33 PMUsing OpenAI default providers
Using OpenAI default providers
Using OpenAI default providers
Provider ollama:chat:command-r:latest is not a valid embedding provider for 'answer relevancy check', falling back to default
Using OpenAI default providers
Note that I am not using any OpenAI default providers (checked in my OpenAI console to make sure I was not using any model)elsatch
08/21/2024, 3:38 PMelsatch
08/21/2024, 3:51 PMdescription: "Gemma generator - Command-R grader"
prompts:
- file://prompts/prompt_baseline.txt
- file://prompts/prompt_xml.txt
providers:
- id: ollama:chat:gemma2:2b-instruct-q8_0
defaultTest:
options:
provider:
text:
id: ollama:chat:gemma2:2b-instruct-q8_0
label: gemma2_2b
config:
temperature: 0
stream: "false"
system: "You are an expert journalist that created compeling stories about local events"
embedding:
id: ollama:embeddings:nomic-embed-text:latest
assert:
- type: context-relevance
threshold: 0.8
- type: context-faithfulness
threshold: 0.8
- type: answer-relevance
threshold: 0.8
tests:
- description: Event question
vars:
query: "What event happened a few years ago at Fablab Alicante?"
context: file://golden_queries_documents/q15.txt
options:
provider: ollama:chat:command-r:latest
But got the error: Provider ollama:chat:command-r:latest is not a valid embedding provider for 'answer relevancy check', falling back to default
So, setting the provider for text at test level, without specifying it's a text provider, will block the use of the embeddings defined at default test level.elsatch
08/21/2024, 3:55 PMyaml
description: "Gemma generator - Command-R text grader"
prompts:
- file://prompts/prompt_baseline.txt
- file://prompts/prompt_xml.txt
providers:
- id: ollama:chat:gemma2:2b-instruct-q8_0
defaultTest:
options:
provider:
text:
id: ollama:chat:gemma2:2b-instruct-q8_0
label: gemma2_2b
config:
temperature: 0
stream: "false"
system: "You are an expert journalist that created compeling stories about local events"
embedding:
id: ollama:embeddings:nomic-embed-text:latest
assert:
- type: context-relevance
threshold: 0.8
- type: context-faithfulness
threshold: 0.8
- type: answer-relevance
threshold: 0.8
tests:
- description: Event question
vars:
query: "What event happened a few years ago at Fablab Alicante?"
context: file://golden_queries_documents/q15.txt
options:
provider:
text:
id: ollama:chat:command-r:latest
Will it mask the text provider defined at defaultTest level? Nope, it will fail complaning that command-R is not a valid embeddings provider:
Error: Invalid provider definition for output type 'embedding': {
"text": {
"id": "ollama:chat:command-r:latest"
}
}
Error: Invalid provider definition for output type 'embedding': {
"text": {
"id": "ollama:chat:command-r:latest"
}
}
at getGradingProvider (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:82:19)
at getAndCheckProvider (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:91:35)
at matchesAnswerRelevance (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:396:37)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async runAssertion (/usr/local/lib/node_modules/promptfoo/dist/src/assertions.js:881:17)
at async /usr/local/lib/node_modules/promptfoo/dist/src/assertions.js:1138:24elsatch
08/21/2024, 4:37 PMdescription: "Gemma generator - Command-R text grader plus embeddings"
prompts:
- file://prompts/prompt_baseline.txt
- file://prompts/prompt_xml.txt
providers:
- id: ollama:chat:gemma2:2b-instruct-q8_0
defaultTest:
options:
provider:
text:
id: ollama:chat:gemma2:2b-instruct-q8_0
label: gemma2_2b
config:
temperature: 0
stream: "false"
system: "You are an expert journalist that created compeling stories about local events"
embedding:
id: ollama:embeddings:nomic-embed-text:latest
assert:
- type: context-relevance
threshold: 0.8
- type: context-faithfulness
threshold: 0.8
- type: answer-relevance
threshold: 0.8
tests:
- description: Event question
vars:
query: "What event happened a few years ago at Fablab Alicante?"
context: file://golden_queries_documents/q15.txt
options:
provider:
text:
id: ollama:chat:command-r:latest
embedding:
id: ollama:embeddings:nomic-embed-text:latestelsatch
08/21/2024, 5:19 PMyaml
description: "Gemma generator - Command-R text grader plus embeddings - assets"
prompts:
- file://prompts/prompt_baseline.txt
- file://prompts/prompt_xml.txt
providers:
- id: ollama:chat:gemma2:2b-instruct-q8_0
defaultTest:
options:
provider:
text:
id: ollama:chat:gemma2:2b-instruct-q8_0
label: gemma2_2b
config:
temperature: 0
stream: "false"
system: "You are an expert journalist that created compeling stories about local events"
embedding:
id: ollama:embeddings:nomic-embed-text:latest
assert:
- type: context-relevance
threshold: 0.8
options:
provider:
text:
id: ollama:chat:command-r:latest
embedding:
id: ollama:embeddings:nomic-embed-text:latest
- type: context-faithfulness
threshold: 0.8
options:
provider:
text:
id: ollama:chat:command-r:latest
embedding:
id: ollama:embeddings:nomic-embed-text:latest
- type: answer-relevance
threshold: 0.8
options:
provider:
text:
id: ollama:chat:command-r:latest
embedding:
id: ollama:embeddings:nomic-embed-text:latest
tests:
- description: Event question
vars:
query: "What event happened a few years ago at Fablab Alicante?"
context: file://golden_queries_documents/q15.txtelsatch
08/21/2024, 5:20 PMarseni_24643
08/21/2024, 6:29 PMyaml
description: 'minimal repo'
prompts:
- 'test prompt {{input}}'
providers:
- "openai:chat:gpt-4o"
defaultTest:
assert:
- type: llm-rubric
value: 'Check if response is good'
tests:
vars:
- input: my test
Running: npx promptfoo@latest eval --grader openai:gpt-3.5-turbo --no-cache
I get:
TypeError: finalProvider.callApi is not a function
TypeError: finalProvider.callApi is not a function
at matchesLlmRubric (/home/arseni/.npm/_npx/1754c693ca488390/node_modules/promptfoo/dist/src/matchers.js:249:38)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async runAssertion (/home/arseni/.npm/_npx/1754c693ca488390/node_modules/promptfoo/dist/src/assertions.js:846:17)
at async /home/arseni/.npm/_npx/1754c693ca488390/node_modules/promptfoo/dist/src/assertions.js:1138:24arseni_24643
08/21/2024, 7:24 PMyaml
defaultTest:
options:
provider: openai:gpt-3.5-turbo
tests:
- description: Evaluate output using LLM
assert:
- type: llm-rubric
value: Is written in a professional tone
When the "provider" should be indented by 1 level to be parsed correctly. I'm thinking the "rubric prompt" text later on probably has the same issue but I haven't tested thatian
08/22/2024, 6:49 AMian
08/22/2024, 6:51 AMian
08/22/2024, 7:00 AMelsatch
08/22/2024, 12:55 PMarseni_24643
08/22/2024, 4:05 PMelsatch
08/24/2024, 11:26 PMian
08/26/2024, 7:56 PMian
08/26/2024, 7:56 PM