https://promptfoo.dev/ logo
Disable provider, programmatic testing only
# ❓|questions
i
I'd like to be able to use promptfoo optionally as a library for running manual tests on LLM outputs which have already been produced. So I have a dataset of 10000s of LLM outputs and need to sample test their accuracy and output format, it's basically a multilabelling pipeline (the output from the LLM is an array of N values from a discrete list). Is this the right library for my use case? 🤔 we may use the ai as a judge elements in the future but we're just starting with the programmatic testing for the moment.
Turns out you can just have "- echo" as the provider, which means I can produce a json file which contains my data and use default tests cases like is-json