https://promptfoo.dev/ logo
Defining providers in csv
# ❓|questions
p
when loading tests from a csv under the __expected column, how do we define providers if we need model-graded metrics? in the yaml normally I can just define providers like this:
Copy code
assert:
      - type: llm-rubric
        value: ensure that the response is relevant to the question
        provider:
          id: openai:gpt-4
          config:
            apiKey: API_KEY
            apiBaseUrl: BASE_URL
but not sure how to properly define providers in the csv I pass in. the docs only show how to define the metrics themselves.
to add - i'd prefer to use a csv rather than defining everything in a yaml cause I have over 1000 test cases
ah nevermind, seems that I can just set some env variables to get the job done
i
👍 this is the way to go for CSV, which is a bit more limited in terms of configuration - lmk if you have any other blockers
p
semi-related, but how do we properly define custom python metrics in a csv? following the pattern in the docs, this is what im inputting into the
__expected
column in my csv
python: file://custom_relevance.py
but when I run the eval it comes back with
Python code execution failed: SyntaxError: invalid syntax
interestingly, I noticed that when I define a custom metric in my csv, the type and value of the metric correctly populates in the ui despite the error. https://cdn.discordapp.com/attachments/1229828690322395136/1230612624291397672/Screenshot_2024-04-18_at_4.15.06_PM.png?ex=6633f451&is=66217f51&hm=e50d5a5f8dafb7d551320cd444f2832f8dd3c51a572ad086dbf3d1043c34f07a&
i
I'm able to repro. It looks like if you remove the space
python:file://custom_relevance.py
it will run fine 😱 I've put in a fix
p
oh wow, yeah that works! and thanks for the quick fix