douar
12/27/2023, 10:40 PMpavirp
12/28/2023, 12:45 PMdouar
01/02/2024, 3:14 PMian
01/03/2024, 4:07 PMassertions module, or implement a thin [custom provider](https://promptfoo.dev/docs/providers/custom-api) or [custom script](https://promptfoo.dev/docs/providers/custom-script) that just returns your static output.
That said, I'm happy to brainstorm ways to support this with the CLI. What's your ideal way for the CLI to ingest LLM outputs?ian
01/03/2024, 6:48 PMpromptfoo eval --assertions assert.yaml --llm-outputs outputs.json
assert.yaml is a list of assertions (similar to the regular config), and outputs.json is a JSON array of strings. It would run the assertions on each output.
LMK what you thinkdouar
01/03/2024, 6:58 PMoutputs.json file, correct? And that file is what the --assertions assert.yaml file would run its assertions on?ian
01/03/2024, 7:01 PMoutputs.json yourself, with whatever LLM outputs you want to testdouar
01/03/2024, 7:03 PMian
01/03/2024, 7:25 PMdouar
01/03/2024, 7:56 PMdouar
01/04/2024, 3:08 PMdouar
01/08/2024, 3:10 PM