https://promptfoo.dev/ logo
Can't use .yaml file for provider config
# ❓|questions
n
Hey ! I'm starting to wonder if it's even possible. I've tried quite a few things. Both of the following have been tried - with/without
file://
-> this does not result in an error, the
id:
is correctly taken into account, however none of the other parameters are. - with `yaml`/`yml` -> straight up an error : `TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".yml" for /app/test/providers/gemma.t_0.tokens_1024.yml`*
Copy code
yaml
providers: [
  file://./providers/phi3.t_0.tokens_1024.yml
]
Copy code
yaml
providers:
  - file://./providers/phi3.t_0.tokens_1024.yml
File content :
Copy code
yaml
id: ollama:chat:phi3
prompts:
  - phi3_template_prompt
config:
  temperature: 0
  max_tokens: 1024
*Full error : https://cdn.discordapp.com/attachments/1265076695765880916/1265076696365662290/image.png?ex=66a031fc&is=669ee07c&hm=53506ccddddbf95af0f2950e4f4773277820494a51adcc66bd68d36c4754a32a&
i
hmmm... can you try something like
Copy code
yaml
- id: file://path/to/file.yml
  label: phi3_tokens_2014
v
I am hitting a similar error, unable to use a file for providers.. I don't want to redefine providers section in every eval test file
i
providers support file imports, so if you can provide an example configuration that allows me to reproduce the issue that would be excellent
v
i
hmm - try with yaml first
I did it with yaml recently, not sure about json... looking at the code..
v
Sure thank you 🙂
Error: Invariant failed: Provider config ./provider.yaml must have an id provider.yaml file contents
Copy code
- id: "openai:chat:gpt-4-turbo"
  label: "GPT-4 Turbo"
  config:
    temperature: 0
I must be missing something
i
try it without the
-
and unindent
the file is read as a single provider
(not a list)
v
Copy code
description: 'Consultation Eval'

prompts: 
  - file://lmn_chat.json
 
providers:
  - file://./provider.yaml
i
yep. you have to update
provider.yaml
to make it just id, label, and config keys without the list prefix
-
v
Thanks that worked 🥳
i will yaml then
Not too hung up on json,
Thank you
i
v
@ian - Hola, If i have want to use multiple providers, should i list them like this?
Copy code
providers:
  - file://../../list.yaml
  - file://../../provider1.yaml
i
That's correct, right now you would have to list each separately
n
I've not been able to make it work on my side... Not sure about what i'm doing wrong. In my case, it seems like the var
prompts
than you can use to filter which prompts are or aren't used by the providers does not work. Current output : - phi3 model X phi3 template prompt - phi3 model X gemma template prompt - gemma model X phi3 template prompt - gemma model X gemma template prompt Excepted output : - phi3 model X phi3 template prompt - gemma model X gemma template prompt config.yml
Copy code
yml
# We define the models prompt templates, used to structure the whole prompt.
# The tool iterate over those, and for each one, will add iteration depending on the variables / providers to use.
prompts:
  - id: "{{phi3_template_prompt}}"
    label: phi3_template_prompt
  - id: "{{gemma_template_prompt}}"
    label: gemma_template_prompt

# Providers are the models that will be used to answer the prompts.
providers:
  - file://./providers/phi3.t_0.tokens_1024.yml
  - file://./providers/gemma.t_0.tokens_1024.yml

# Global variables and assertions that are used in all tests.
defaultTest:
  vars:
    # Template prompts source files, then used in the `prompts` section.
    phi3_template_prompt: file://./prompts/model-templates/phi3.txt
    gemma_template_prompt: file://./prompts/model-templates/gemma.txt
    # System prompts source files, then used in the `providers` section.
    system_prompt_list: [
      file://./prompts/agent-system-prompts/api-event-expert/phi3.v001.txt,
      file://./prompts/agent-system-prompts/dumb.txt
    ]
gemma.t_0.tokens_1024.yml
Copy code
yml
id: ollama:gemma:2b
label: "[gemma:2b] | temperature 0 | tokens 1024"
prompts:
  - gemma_template_prompt
config:
  temperature: 0
  max_tokens: 1024
phi3.t_0.tokens_1024.yml
Copy code
yml
id: ollama:chat:phi3:mini
label: "[phi3:mini] | temperature 0 | tokens 1024"
prompts:
  - phi3_template_prompt
config:
  temperature: 0
  max_tokens: 1024
--------- Hey @vparthay 🙂 Have you had a case where you are using a variable inside the provider file ? I'm wondering if this might be the problem 🤔 The label is correctly loaded and i do see this in the logs :
Copy code
2024-07-25 11:47:29 Loaded provider ollama:gemma:2b from ./providers/gemma.t_0.tokens_1024.yml
2024-07-25 11:47:29 Loaded provider ollama:gemma:2b from ./providers/gemma.t_0-50.tokens_1024.yml
Hello @ian, should i open a bug issue regarding this ?
i
yeah that would be great! Easier to keep track of genuine bugs that way
2 Views