https://promptfoo.dev/ logo
evaluation/assertions only
# ❓|questions
d
Is this possible with promptfoo? I already have an output from an LLM and I just want to run the output against promptfoo assertions, is this possible?
p
+1: I have the same question
i
Can you tell me a little more about your setup? One way would be to use the
assertions
module exported by promptfoo (for example: ) Or do you want to do this from the command line, or using a different programming language?
d
Hi @ian , Happy New Years! I am using github action to automate a few things. The custom provider I am using is only accessible internally at the moment via a python script, but I'd like to pass the provider's output to promptfoo for assertions. So I won't need promptfoo to call the provider as I can do so internally but I would like promptfoo to perform its assertions on the output and receive the pass/fail results from promptfoo. If I can do so from the command line I think that may help a lot given that I am using github action to automate. Is that possible or would the assertions module be my best route at this time?
i
Right now there isn't a direct way to do this with the CLI. Your options are to either use the
assertions
module, or implement a thin [custom provider](https://promptfoo.dev/docs/providers/custom-api) or [custom script](https://promptfoo.dev/docs/providers/custom-script) that just returns your static output. That said, I'm happy to brainstorm ways to support this with the CLI. What's your ideal way for the CLI to ingest LLM outputs?
Here's what I'm thinking of implementing:
Copy code
promptfoo eval --assertions assert.yaml --llm-outputs outputs.json
assert.yaml
is a list of assertions (similar to the regular config), and
outputs.json
is a JSON array of strings. It would run the assertions on each output. LMK what you think
d
That setup definitely looks like something that would help resolve the blocker I am having! If I am understanding correctly, the provider's output would be in the
outputs.json
file, correct? And that file is what the
--assertions assert.yaml
file would run its assertions on?
i
that's correct. you'd create
outputs.json
yourself, with whatever LLM outputs you want to test
d
Cool! In that case I'd like to try it out if it's possible? 😄
i
I'll get on it 🙂
d
Thank you!!
i
PR here: . This functionality will be in the next release, although I consider it experimental at this point (open to your feedback once you use it)
d
Sounds good, thank you so much! 😄
i
This change is published in 0.35.0. Here's an example:
Copy code
promptfoo eval --assertions asserts.yaml --model-outputs outputs.json
d
Thank you! Will be implementing it soon and provide any feedback I have 🙂
3 Views