https://promptfoo.dev/ logo
Python var loader not working in GitHub Actions CI
# ❓|questions
s
Hi! I'm running into an issue while using promptfoo in a GitHub Action CI pipeline triggered by a Pull Request. Here's the setup: In my PR, I have a test config file like this: prompts: - '{{query}}' providers: - python:rag_provider.py:call_api defaultTest: assert: - type: answer-relevance threshold: 0.7 - type: context-relevance threshold: 0.7 vars: context: python:rag_context_loader.py:get_var tests: - question.csv The intention is for the context variable to be dynamically loaded by executing the Python function get_var() inside rag_context_loader.py. However, during the GitHub Action run (CI triggered by the PR), promptfoo does not seem to execute the Python function. Instead, it sets the context to the raw string "python:rag_context_loader.py:get_var".
i
Can you confirm the version or Promptfoo running in CI?
s
In my CI, promptfoo is installed via npm. Here's the relevant part of the workflow file: - name: Install promptfoo & login run: | npm install -g promptfoo promptfoo auth login --host https://www.promptfoo.app --api-key ${{ secrets.PROMPTFOO_API_KEY }} So I think the version installed is the latest one from npm at runtime