https://promptfoo.dev/ logo
building multiple choice eval
# ❓|questions
p
hello, i'm building a multiple-choice general purpose (such as mmlu) eval in our local language (turkish) as an academic paper. boilerplate part is mostly finished and i need a proper way to evaluate different models (gpt-4o, 3.5 sonnet, llama etc.). we're planning to use gpt-4o as a judge for now but we may need to manually judge it too. we have some questions ready in format of {question, category, [a,b,c,d,e answers], correct answer}. is promptfoo a nice fit for our requirements
i
just to confirm, is the idea that your prompt will include {question, category, answers} and you want to check that the output contains {correct answer}?
p
yeah, basically.
{"what's 2 + 2?", "math", [2, 4, 6, 8, 10], B}
can swap
B
with
4
anyway, whatever works easier.
i
cool, yeah this should work well with promptfoo if you want to keep it all in CSV, you can use the [CSV format](https://promptfoo.dev/docs/configuration/parameters#import-from-csv) for tests. You'd have to rename the columns to match the
{{variableName}}
you use in the prompt, and then use the
__expected
column to set the expected correct answer output. I assume your prompt will do something like ask it to output the multiple choice answer, so you can use
starts-with:B
in the
__expected
column for example. Also happy to walk you through the yaml config, which is more flexible than CSV (may be necessary if you're doing something more complicated)