https://promptfoo.dev/ logo
human evals?
# ❓|questions
a
Howdy! I'm liking promptfoo for some quick and dirty LLM comparisons. I'm wondering if promptfoo has any plans to allow humans to "take the same test" for benchmarking purposes. Or, if anyone knows a tool that would work well for that, I'm all ears! Currently I'm struggling though building out something good in airtable, and I'm also starting to look at the cost of building out a Django app, unfortunately.
i
I think that's a really cool idea. To be honest I haven't thought about how to build this sort of flow into promptfoo. One idea for partially solving this is you could turn the test cases into a google form, and eval them as standalone outputs
a
Oh nice idea. I don't know how to create a dynamic form. Let's say I have 100 questions. I'd like a single form to ask them one at a time, or people could lose their progress. Not sure how to do that with sheets. I can do it with airtable interfaces but I'd love to see something more directly in promptfoo one day! Cheers