https://promptfoo.dev/ logo
Trouble with per-provider prompt setup in `tests.y...
# ❓|questions
a
Hello, we have found that both today, and last time we tried this ~3 weeks ago, that we were unable to get promptfoo to run successfully when using per-provider prompts, as per the documentation. Example setup below:
Copy code
yaml
description: 'Example'

tests: data.csv

prompts:
  - id: file://prompts_workflow.txt
    label: workflow_prompts
  - id: file://prompts_dag_resolver.txt
    label: resolver_prompts

providers:
  - id: 'python:../../../../shims/shim_ll_workflow_api.py'
    label: 'Basic Traffic Light Workflow: wp_ll_workflow_api_Payment_Period'
    prompts:
      - workflow_prompts
    config:
      workflow_id: '11'
      inputs:
        rag_descriptor: |-
          It should always be grey.
        one_line_format: Brief, concise, one paragraph of no more than 3-4 sentences.
        answer_summary_format: Markdown-formatted response.
      evaluateOptions.maxConcurrency: 4
      evaluateOptions.repeat: 1
      evaluateOptions.delay: 100
      evaluateOptions.showProgressBar: true
Enabling debugging suggests that promptfoo is finding and enumerating the prompts, but then it doesn't seem to find any testcases to run, returning 0 successes and 0 failures. This occurs regardless of whether the provider prompt files contain a single prompt or multiple prompts separated by
---
. Removing the per-provider prompt fields and reverting to a single
prompts.txt
file resolves the issue. Appreciate any guidance you may be able to provide!
i
👀 thanks for the report. will look into this tomorrow (it's about 1am here)
hey, just letting you know @michaelmichaelmichael and I are working on it. This area of the code is surprisingly complex unfortunately, but we'll have a fix pretty soon
m
Hi @alexclaydon Apologies for the delay. We deployed a fix for this in 0.67.0 along with an example: https://github.com/promptfoo/promptfoo/tree/0.67.0/examples/prompt-labels Please let me know if you run into any other issues!