https://promptfoo.dev/ logo
any way to provide custom providers while using go...
# ❓|questions
d
I intend to use promptfoo to run evaluation of multiple test cases which have been stored in a google sheet and while trying to write the assert in the __expected column I noticed there was no mention on how to use a custom provider for evaluation of model-based metrics i intend to use ollama:llama3.1 but it defaults to using chatgpt
and for the similarity metric as well I would like to use a custom provider
m
@dev_nerd_41284 can you share your promptfoo config? For custom providers in model graded metrics, see https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader You can apply this to any model graded metric but be careful as smaller and older models often do not work as well
d
Yeah, I've been able to do this when I'm using the config file to write assertions but I'm asking for a way for me to able to do the same while working with a google sheets file as I can't add a asserion.provider metric while using google sheets
description: "RAG - End to end test with comparison" prompts: - prompts.json providers: - id: customProvider.js tests: https://docs.google.com/spreadsheets/d/1jGx8Ag3mSTrFj7P4qJ2KGf_WNXskvd7vlWIUaPGdcg0/edit?usp=sharing
in some test cases I wish to use a llm rubric assertion and in some I wish to use an embedder model, it was quite convenient to do this in the config file but I am unable to understand a way to do this for google sheets
description: "RAG - End to end test with comparison" prompts: - prompts.json providers: - id: customProvider.js tests: - vars: filePath: Question - What is the difference between supervised and unsupervised learning in machine learning? noOfvariants: 1 application: AGA assert: - type: llm-rubric provider: ollama:llama3.1 value: Content after Comprehensiveness: makes sense in the context of comprehensivness with respect to the Question-Answer pair defined by 'Question -' and 'Answer -' which comes before Correctness:, always return in JSON format - vars: filePath: Question - What is the difference between supervised and unsupervised learning in machine learning?. noOfvariants: 1 application: answergen assert: - type: similar value: 'Supervised learning involves training a model on labeled data, where each input is paired with a corresponding output.' provider: ollama:embeddings:bge-large this is the kind of structure I want to replicate in google sheets
m
@dev_nerd_41284 can you try to add a defaultTest block like this? https://www.promptfoo.dev/docs/usage/troubleshooting/#openai-api-key-is-not-set
d
sure I'll try this
alright this works thanks a lot for your prompt responses
m
you're welcome! I'll update the test docs