https://promptfoo.dev/ logo
Comparison with other projects
# ❓|questions
k
I'm looking into using promptfoo, and I really like what I see so far! However, I would love to compare it with other similar projects so that I'm sure I'm using the right tool for the job. If it's not out of line, I'd love to hear what projects compete with promptfoo and why I'd use one versus the other. To start, there's a huge list here of potential other projects to consider: https://github.com/tensorchord/Awesome-LLMOps?tab=readme-ov-file#llmops
i
there are sooo many eval options! I keep my own list here: https://www.ianww.com/llm-tools
in general, people who prefer promptfoo like it because: 1. It runs locally with no cloud dependency (no logins, privacy issues, you own all the evals, etc) 2. It's language agnostic (most others are python only and require you to instrument your app code) 3. It abstracts code vs evals - there's a separation of concerns between the yaml config, which defines the eval, and the rest of your code
that's my 2c - i'm sure there are many other reasons depending on what matters to you
k
This is really helpful! Off the top of your head, what are a few of the most notable ones you'd use if promptfoo didn't exist?
i
what kind of app are you building and do you care about observability/traces in production?
k
Adding AI features, including a chatbot, to an existing product. I think yes to your second question, though I don't have a good idea of exactly what that looks like.
> 1. It runs locally with no cloud dependency (no logins, privacy issues, you own all the evals, etc) I think this is huge. Do you know of any others with this model, or are most/all of the others necessarily running on the cloud?
e
Things I like about Promptfoo: - Easiest to get started - Great for A/B testing generations - Flexible and modular - Allows local models out of the box - Good integrations and detailed ways to create your own - Same tool can be used for evals and red teaming - Several metrics for RAG available out of the box - Very responsive developers - You can start getting vañue out of it in a few hours - The interface is slick - And it doesn't try to make you send your data to their cloud or SaaS solution all the time, everywhere
Thanks for the list, it's really useful. I just added Giskard.ai to the list, with an estimated funding valuation after securing a second round.
I think they have several cool ideas in the pipeline (but it's quite less modular and simple compared to promptfoo)
k
Friendly ping on this specific question, @ian, if you happen to know. Thanks!
m
@kentyman I like to think that we are the best one! We are the most popular and we are deeply committed to building the best possible eval experience in open source. There are a few tools inspired by promptfoo such as https://evals.tbuckley.com/documentation Happy to discuss promptfoo on a call if you have more questions https://cal.com/michael-dangelo/30min
k
@michaelmichaelmichael That's extremely generous of you! I scheduled for tomorrow.
@michaelmichaelmichael, thanks again for meeting. You mentioned something about OpenAI dropping their own eval tools in favor of Promptfoo. Do you have a link that talks about that?
m
Unfortunately, that's not quite true (I wish it was). The Solutions team and Developer Experience team at OpenAI use Promptfoo in demos with their customers (many of the people in this discord found out about promptfoo from OpenAI), and OpenAI stopped regularly updating their eval tool (https://github.com/openai/evals) but we are not officially affiliated.