https://promptfoo.dev/ logo
image / vision support
# ❓|questions
c
Asking because I didn't see any documentation on this topic. Is there support for using images in gpt-4-vision or claude evals?
I don't see any reason reading some of the code that it wouldn't, given the JSON stuff is basically just doing
JSON.parse
I'm gonna give it a try and report back
hm, gpt-4-vision-preview seems to work out of the box while anthropic's image stuff is (of course) non-standard and doesn't seem to work. I hope PRs are accepted...
https://github.com/promptfoo/promptfoo/pull/637 documentation for openai vision + an example
i'll try to make anthropic work too while i'm at it
i
hey @camdenc, really appreciate you looking into this. PR looks good, just merged it
incidentally promptfoo also supports dall-e.. just not really documented at this point
c
for sure, thanks!
I just opened one for supporting vision prompting in anthropic https://github.com/promptfoo/promptfoo/pull/639
now tool use in claude will be a bit more challenging I think
I opened a ticket about tool use, it was just released in beta. Anthropic has chosen some annoyingly different paths here. I'm going to give some thought on how to approach it. but broadly how are you thinking about the OpenAI prompt format vs all the others? I'm kinda thinking that OpenAI's prompt format should be the lingua franca that any other provider can interpret (assuming 1-1 functionality etc.). But each provider can also support its own prompt format as well.
i
👍 I responded on github, but yep I agree with you! Thanks so much for helping out with this