Hi! I'm running into an issue while using promptfoo in a GitHub Action CI pipeline triggered by a Pull Request.
Here's the setup:
In my PR, I have a test config file like this:
prompts:
- '{{query}}'
providers:
- python:rag_provider.py:call_api
defaultTest:
assert:
- type: answer-relevance
threshold: 0.7
- type: context-relevance
threshold: 0.7
vars:
context: python:rag_context_loader.py:get_var
tests:
- question.csv
The intention is for the context variable to be dynamically loaded by executing the Python function get_var() inside rag_context_loader.py.
However, during the GitHub Action run (CI triggered by the PR), promptfoo does not seem to execute the Python function. Instead, it sets the context to the raw string "python:rag_context_loader.py:get_var".