neoteristis
07/22/2024, 10:43 PMfile:// -> this does not result in an error, the id: is correctly taken into account, however none of the other parameters are.
- with `yaml`/`yml` -> straight up an error : `TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".yml" for /app/test/providers/gemma.t_0.tokens_1024.yml`*
yaml
providers: [
file://./providers/phi3.t_0.tokens_1024.yml
]
yaml
providers:
- file://./providers/phi3.t_0.tokens_1024.yml
File content :
yaml
id: ollama:chat:phi3
prompts:
- phi3_template_prompt
config:
temperature: 0
max_tokens: 1024
*Full error :
https://cdn.discordapp.com/attachments/1265076695765880916/1265076696365662290/image.png?ex=66a031fc&is=669ee07c&hm=53506ccddddbf95af0f2950e4f4773277820494a51adcc66bd68d36c4754a32a&ian
07/23/2024, 7:42 PMyaml
- id: file://path/to/file.yml
label: phi3_tokens_2014vparthay
07/25/2024, 5:53 AMian
07/25/2024, 6:11 AMvparthay
07/25/2024, 6:13 AMian
07/25/2024, 6:14 AMian
07/25/2024, 6:14 AMvparthay
07/25/2024, 6:16 AMvparthay
07/25/2024, 6:17 AM- id: "openai:chat:gpt-4-turbo"
label: "GPT-4 Turbo"
config:
temperature: 0
I must be missing somethingian
07/25/2024, 6:18 AM- and unindentian
07/25/2024, 6:18 AMian
07/25/2024, 6:18 AMvparthay
07/25/2024, 6:18 AMdescription: 'Consultation Eval'
prompts:
- file://lmn_chat.json
providers:
- file://./provider.yamlian
07/25/2024, 6:19 AMprovider.yaml to make it just id, label, and config keys without the list prefix -vparthay
07/25/2024, 6:19 AMvparthay
07/25/2024, 6:19 AMvparthay
07/25/2024, 6:19 AMvparthay
07/25/2024, 6:19 AMvparthay
07/25/2024, 7:25 AMproviders:
- file://../../list.yaml
- file://../../provider1.yamlian
07/25/2024, 8:18 AMneoteristis
07/25/2024, 9:55 AMprompts than you can use to filter which prompts are or aren't used by the providers does not work.
Current output :
- phi3 model X phi3 template prompt
- phi3 model X gemma template prompt
- gemma model X phi3 template prompt
- gemma model X gemma template prompt
Excepted output :
- phi3 model X phi3 template prompt
- gemma model X gemma template prompt
config.yml
yml
# We define the models prompt templates, used to structure the whole prompt.
# The tool iterate over those, and for each one, will add iteration depending on the variables / providers to use.
prompts:
- id: "{{phi3_template_prompt}}"
label: phi3_template_prompt
- id: "{{gemma_template_prompt}}"
label: gemma_template_prompt
# Providers are the models that will be used to answer the prompts.
providers:
- file://./providers/phi3.t_0.tokens_1024.yml
- file://./providers/gemma.t_0.tokens_1024.yml
# Global variables and assertions that are used in all tests.
defaultTest:
vars:
# Template prompts source files, then used in the `prompts` section.
phi3_template_prompt: file://./prompts/model-templates/phi3.txt
gemma_template_prompt: file://./prompts/model-templates/gemma.txt
# System prompts source files, then used in the `providers` section.
system_prompt_list: [
file://./prompts/agent-system-prompts/api-event-expert/phi3.v001.txt,
file://./prompts/agent-system-prompts/dumb.txt
]
gemma.t_0.tokens_1024.yml
yml
id: ollama:gemma:2b
label: "[gemma:2b] | temperature 0 | tokens 1024"
prompts:
- gemma_template_prompt
config:
temperature: 0
max_tokens: 1024
phi3.t_0.tokens_1024.yml
yml
id: ollama:chat:phi3:mini
label: "[phi3:mini] | temperature 0 | tokens 1024"
prompts:
- phi3_template_prompt
config:
temperature: 0
max_tokens: 1024neoteristis
07/25/2024, 10:01 AM2024-07-25 11:47:29 Loaded provider ollama:gemma:2b from ./providers/gemma.t_0.tokens_1024.yml
2024-07-25 11:47:29 Loaded provider ollama:gemma:2b from ./providers/gemma.t_0-50.tokens_1024.ymlneoteristis
07/26/2024, 11:33 AMian
07/26/2024, 5:04 PM