https://promptfoo.dev/ logo
Managing file paths with a custom python provider
# ❓|questions
k
So Im writing my agent service that has multiple prompts. I want to write a eval for each of these. The pormptfooconfig.yaml file will also be different for each prompt. Here is an example description: "Invoice Classification" providers: - id: 'file:///../../../../../core/evals/promptfoo_provider.py' label: "claude-3.5-sonnet" config: model_name: "claude-3.5-sonnet" - id: 'file:///../../../../../core/evals/promptfoo_provider.py' label: "gpt-4o-mini" config: model_name: "gpt-4o-mini" evaluateOptions: repeat: 1 prompts: ./invoice_classification_prompt.txt tests: test*.yaml Now the problem with this is, the ID needs to be a relative path, and it’s annoying to maintain that accross multiple config files while my provider path is always the same. What has worked is putting the absolute path eg User/name/path/to/repo But this does not work in my CI/CD. Any way i can solve this?
d
Why not set the absolute path as env variable in CI/CD and then grab that in your ci script and drop it in promptfoo.config using a bash one-liner? This way, you can just use any PLACEHOLDER in the "id" fields in promptfooconfig, as many times as you like and will have to just replace it before you begin running any promptfoo commands.
Sometihng like this in your CI script for inplace string replacement: sed -i 's/PLACEHOLDER_STRING_FROM_CONFIG/{CI-ENV-VARIABLE-absolute-filepath-to-py-provider}/g' promptfooconfig.yaml