So Im writing my agent service that has multiple prompts. I want to write a eval for each of these.
The pormptfooconfig.yaml file will also be different for each prompt.
Here is an example
description: "Invoice Classification"
providers:
- id: 'file:///../../../../../core/evals/promptfoo_provider.py'
label: "claude-3.5-sonnet"
config:
model_name: "claude-3.5-sonnet"
- id: 'file:///../../../../../core/evals/promptfoo_provider.py'
label: "gpt-4o-mini"
config:
model_name: "gpt-4o-mini"
evaluateOptions:
repeat: 1
prompts: ./invoice_classification_prompt.txt
tests: test*.yaml
Now the problem with this is, the ID needs to be a relative path, and it’s annoying to maintain that accross multiple config files while my provider path is always the same.
What has worked is putting the absolute path eg User/name/path/to/repo
But this does not work in my CI/CD.
Any way i can solve this?