https://promptfoo.dev/ logo
Are there any best practices to use this framework...
# ❓|questions
a
(like testing llava systems for some tasks...)
i
I haven't tried with Llava, but have gotten it to work with Dalle-3 Example config: Built-in Dall-E provider (in case you need to implement a custom provider): basically it returns Markdown-formatted output,
![caption](image-url)
, and then you can enable Markdown in the webview table settings to view the images
sorry, the above answer assumed you want to view image outputs. if you're sending image inputs, this is probably best done with a custom provider ([javascript](https://promptfoo.dev/docs/providers/custom-api), [python](https://promptfoo.dev/docs/providers/python), or [other](https://promptfoo.dev/docs/providers/custom-script)) that sends the image to the model in whatever required format
c
Hey @ian ! What an awesome tool, you've done an excellent job building it. I'm prototyping using promptfoo to do comparisons of some visual LLMs. I rely entirely on the web viewer. My workflow is as follows: Requirements: 1. Compare two different models, each hosted in a separate environment, accessible via APIs 2. Both models require an image as input 3. Eval against 100 different test input images Solution: 1. I use the input image URL as a variable, so I have 100 variables 2. I use the prompt as a JSON representing the request body to the APIs 3. Two custom python providers, one for each model, which call out to the APIs 4. I use latency as an automatic assertion 5. I use the thumbs up/down as manual assertions to compare image quality See the photo I attached - Is there any way to view a variable as an image in the web viewer? I tried markdown but no luck https://cdn.discordapp.com/attachments/1204780043436560394/1241137132148363386/Screenshot_2024-05-17_at_1.26.09_PM.png?ex=66491a87&is=6647c907&hm=50779698f478caef72c053a5a7cf41d5bbccbe6430af54e083c6002339b85170&
i
That's currently not supported, but I can definitely make that change 👀
c
That would be awesome! Let me know if you need me to create an issue or anything. Also indifferent on how the image appears, whether markdown or some other mechanism! Whatever is easiest
i
it'll be in the next release