https://promptfoo.dev/ logo
what is the recommended input file type?
# ❓|questions
e
There are currently a lot of file types supported for prompt file. We wonder is there any recommended/better one among all of these? Txt, JSON, js, py, yaml? We realize yaml files sometimes don’t work as expected. So We wonder if maybe there some existing evaluation done , and we don’t have to re-experiment on which file works the best.
i
I usually go with
txt
or
json
. Js or py only necessary if you need to dynamically set input
e
Thank you so much for the prompt response ! What about csv?
i
Csv for tests, not prompts is well supported
On the prompt side, everything is ultimately sent as a json request to the LLM API. If a json prompt is provided, it is sent verbatim. Otherwise, for chat completion endpoints, the raw text is wrapped as a user message.
e
Thank you @ian , you are the best
Hmmm I think I probably wasn’t clear, I’m sorry. I realize there’re multiple references to the ‘prompt’. I meant test input data.
i
ohh I see. For test input data, I usually do YAML
CSV, JSON are supported. CSV is more constrained (e.g. harder to set up more complex evals), and JSON is verbose (but equivalent to YAML)
what sort of issues were you having with yaml?
e
YAML itself is strict in language and lint. Not many users are familiar with that. If asking someone doesn't have experience at all to use the tool, might be easier to be CSV or JSON. But like you said, there are cons for those two as well.
i
Understood. JSON is an alternative and is treated equivalently to YAML in code 🙂
CSV is easier but there are limitations on what you can express, depending on what you're evaluating
e
Exactly! For CSV the limitation is what I highlighted - just one constraint is possible. How do we/you guys get around that limitation?
Or is there a way to get around?
Btw We do have that need.
i
no workaround for that currently - one per testcase