Two quick questions:
1) Is there anyway to utilize test metadata in PromptFoo view? It's one of the properties in Test Case.
2) Is each Prompt Input <> Output pair a dataset? Is there anyway we can associate tests with datasets or have more granular control to organize our tests?
Thank you so much in advance for helping out!
m
michaelmichaelmichael
09/18/2024, 4:56 AM
1. Not yet but we could get something out quickly. Right now that's only used in redteaming. What are you trying to use it for? We've also been playing with the concept of tags that may be useful to you.
2. I am not sure if I fully understand the question. You can associate prompts with providers like https://github.com/promptfoo/promptfoo/blob/main/examples/prompt-labels/promptfooconfig.yaml or split different tests and datasets into different promptfooconfigs. Happy to hop on a call and discuss further if helpful https://cal.com/michael-dangelo/15min
h
House
09/18/2024, 5:32 AM
Thanks for the crazy fast response time!
Re 1, it's more for grouping eval results by tags. No rush on dev, was just trying to see what is out there.
Re 2, I should clarify, I'm talking about the Dataset tab / heading in PromptFoo View.