https://promptfoo.dev/ logo
See full response from errors.
# ❓|questions
w
When using JSON mode with Groq, their API throws the following error: API error: Failed to generate JSON. Please adjust your prompt. See 'failed_generation' for more details., Type: invalid_request_error, Code: json_validate_failed. In Groq's actual response, the error contains a key that shows you what the output was. When I click the magnifying glass to inspect the request, I can only see input/output. Would be really nice for debugging to be able to look at the response! https://cdn.discordapp.com/attachments/1240324512868995204/1240324513418444821/image.png?ex=664625b8&is=6644d438&hm=81075153807a7b01c074483c54e2116c016b8acb575af852aaa6921457ee4c32&
Also I realize I can use a custom provider around Groq (which I just attempted to do), but configuring the YAML it has been a challenge since I'm running typescript files that get compiled down to CJS. Syntax for pointing to the right files is a bit confusing for me. Just stating this as feedback!
i
the cjs vs esm stuff kills me every time 😭
I have wasted hours/days at a time on it... it seems almost impossible to support every configuration
w
haha i actually have it in a really good spot. Basically my use case was wanting to share my js prompt files to my backend inside of my turborepo, that way I could make sure my evals and codebase were running on the same prompts. It works great and I'm happy to share the code / open a PR if it's something you'd want to include in promptfoo docs / quickstarts.
the problem is pointing the config.yamls to the right provider.js lol
that's a different problem though - the issue I'm posting about right now is just that it would be really nice to be able to inspect the whole request/response. I think it'd be useful even outside of this groq use case. Other providers may be returning error messages in different ways or have other kinds of data that we'd want to see inside of the promptfoo dashboard.
also this might be a crazy idea and I haven't looked into bun enough, but do you think bun could be a solution here?
i
yes, I was helping someone with this problem and bun just magically solved everything when they used it as the runtime instead of node
yeah I agree here as well - let me see what I can do
w
Amazing - thanks Ian!
i
one temporary workaround is to run promptfoo with the
LOG_LEVEL=debug
envar or
promptfoo eval --verbose
to see the full request/response
PR here to add full response data for OpenAI-style providers: https://github.com/promptfoo/promptfoo/pull/791
w
oh sick - thank you so much!
@ian just dropped you a small sponshorship on github, thanks so much for your hard work on promptfoo. It's been really awesome for us so far. 🙏
i
wow, thanks so much. I'm glad you are liking promptfoo!!