alexclaydon
06/18/2024, 7:17 AMyaml
description: 'Example'
tests: data.csv
prompts:
- id: file://prompts_workflow.txt
label: workflow_prompts
- id: file://prompts_dag_resolver.txt
label: resolver_prompts
providers:
- id: 'python:../../../../shims/shim_ll_workflow_api.py'
label: 'Basic Traffic Light Workflow: wp_ll_workflow_api_Payment_Period'
prompts:
- workflow_prompts
config:
workflow_id: '11'
inputs:
rag_descriptor: |-
It should always be grey.
one_line_format: Brief, concise, one paragraph of no more than 3-4 sentences.
answer_summary_format: Markdown-formatted response.
evaluateOptions.maxConcurrency: 4
evaluateOptions.repeat: 1
evaluateOptions.delay: 100
evaluateOptions.showProgressBar: true
Enabling debugging suggests that promptfoo is finding and enumerating the prompts, but then it doesn't seem to find any testcases to run, returning 0 successes and 0 failures. This occurs regardless of whether the provider prompt files contain a single prompt or multiple prompts separated by ---. Removing the per-provider prompt fields and reverting to a single prompts.txt file resolves the issue. Appreciate any guidance you may be able to provide!ian
06/18/2024, 7:49 AMian
06/20/2024, 5:29 AMmichaelmichaelmichael
06/27/2024, 10:46 PM