https://promptfoo.dev/ logo
Workflow dispatch for GitHub Action
# ❓|questions
t
Any way to use the promptfoo GitHub Action only with workflow_dispatch event?
I’ve decided to run the evaluation using directly
npx
but I’m having some problems with the cache
Here the GitHub Action:
Copy code
yaml

---
name: Evaluate

on: workflow_dispatch

jobs:
  eval:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4

      - name: Use Node
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Create cache directory
        run: mkdir -p ~/.cache/promptfoo

      - name: Set up promptfoo cache
        id: cache-promptfoo
        uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-v1-${{ hashFiles('**/promptfooconfig.yaml') }}
          restore-keys: |
            ${{ runner.os }}-promptfoo-v1-
            ${{ runner.os }}-promptfoo-

      - name: Check cache status
        run: echo "Cache hit - ${{ steps.cache-promptfoo.outputs.cache-hit }}"

      - name: Runs the promptfoo evaluation
        run: npx promptfoo@latest eval -c promptfooconfig.yaml -o output.json
        env:
          PROMPTFOO_CACHE_PATH: ~/.cache/promptfoo
          PROMPTFOO_DISABLE_TELEMETRY: 1
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PROMPTFOO_FAILED_TEST_EXIT_CODE: 0
I always have the message:
Creating cache folder at ~/.cache/promptfoo
@ian could you check it?
@ian could you review it, please?
Nothing?
m
Try caching the default cache path as well:
Copy code
name: Evaluate

on: workflow_dispatch

jobs:
  eval:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4

      - name: Use Node
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Set up promptfoo cache
        id: cache-promptfoo
        uses: actions/cache@v4
        with:
          path: |
            ~/.cache/promptfoo
            ~/.promptfoo
          key: ${{ runner.os }}-promptfoo-v2-${{ hashFiles('**/promptfooconfig.yaml') }}
          restore-keys: |
            ${{ runner.os }}-promptfoo-v2-

      - name: Runs the promptfoo evaluation
        run: npx promptfoo@latest eval -c promptfooconfig.yaml -o output.json
        env:
          PROMPTFOO_CACHE_PATH: ~/.cache/promptfoo
          PROMPTFOO_DISABLE_TELEMETRY: 1
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PROMPTFOO_FAILED_TEST_EXIT_CODE: 0
this 1. Removes manual cache directory creation (unnecessary) 2. Adds ~/.promptfoo to cache paths