https://promptfoo.dev/ logo
Support for multi-modal evaluation
# ❓|questions
e
I wonder if there is support for multimodal evaluation, now that there are so many new models like Pixtral focused on Q&A over pictures. Maybe this could reuse the file:// handler passing the raw image to the multimodal model.
m
@elsatch Great question / idea. We have a few examples of evaling multi-modal inputs such as https://github.com/promptfoo/promptfoo/tree/main/examples/image-classification and https://github.com/promptfoo/promptfoo/blob/main/examples/claude-vs-gpt-image but no examples of evaling generated images (or audio video etc). Are you thinking about support for model graded image generation?
e
Oh, the image classification example looks very cool. Indeed, it has the tests: element as in:
tests: file://fashion_mnist_sample_base64.csv
That looks quite similar to the dataset option I was talking about in other thread (https://discord.com/channels/1153767083381903389/1267638832908599396/1286219021070368811 ) Given this code I will revisit that idea too.
But going back to multi-modal evaluation, my main use case is using images for RAG, either for caption creation or information retrieval.
So, for example I might use promptfoo to call llava, baklava, moondream and other models using Ollama and compare the outputs. If images can be passed as argument, it becomes quick trivial to check for the main elements in each image in the text output.
Other very powerful case is in-context captioning, providing de multimodal LLM a summary of the text, the previous and posterior chunks and ask to generate a caption. My intuition is that using this method could provide a much more richer in-domain captioning than just raw image captioning.
r
Can I put video as input?
I got it working from the image example
n
@michaelmichaelmichael I also have a great question for you