https://promptfoo.dev/ logo
Allow DefaultTest.options.provider to handle a Pro...
# ❓|questions
d
Minification and bundling has made it challenging for me to have this work as a node package in my deployed lambda. I've altered my implementation now to pass in ApiProviders instead of filepath references and it seems to work for the providers array which uses the loadApiProviders implementation, however I can't replace my default test provider with a custom provider this way as it looks for a string filepath to load. We aren't allow to use keys to access our Azure OpenAI endpoints and need to rely on an OAuth implementation to do so in our deployed solution, so I'm kinda stuck here at the moment getting this as a deployed API solution to evaluate our tests. Would welcome a fix here as it seems like it was commented in the code to be addressed at a certain point. https://github.com/promptfoo/promptfoo/blob/c196b47d221d47cef1460ec4e1279e5c630effb0/src/providers/index.ts#L18 https://github.com/promptfoo/promptfoo/blob/c196b47d221d47cef1460ec4e1279e5c630effb0/src/index.ts#L53
"Error running promptfoo tests: Error: src must be a string or an object describing the source" is what I get when I try to set the defaultTest.options.provider like so: defaultTest: { assert: config.defaultAssertions.map((assertion) => ({ ...assertion, type: assertion.type as AssertType, })), options: { provider: async (prompt: string) => validator.callApi(prompt), },
Any updates here?