https://promptfoo.dev/ logo
Dynamically output context from custom API provide...
# ❓|questions
s
I have a custom API provider that contains not only the output of the prompt but also relevant context such as the data used to answer the prompt, the query used to obtain the data, and examples used to generate the query. I want to be able to use those pieces of context in promptfoo. For example, I might want to use the context-faithfulness metric to test if the generated output used the provided data. It doesn't look like I can use the output dataset context from my custom provider in tests. Is there a way to use additional fields returned from a custom provider in tests? One way around this might be to save the data out to a CSV each run and to reference that CSV file in the tests but I'd rather avoid doing that. Any ideas how I can use these additional pieces of information in my tests?
I suppose I could also use a test option to determine what the output should be - either the final output, SQL query generated, or the data output by that query. I could specify what relevant output I wanted and just use that. It would allow me to get closer to my goals but I don't think I'd be able to test context faithfulness between the final output and the data because I'd only be able to output one at a time. Thoughts @ian ?
i
Hi @sirtylerdurden., sorry for the slow reply (I've been in the hospital with a family member). There isn't a good built-in way to use fields returned by a provider besides the output. However, this sounds like a good candidate for importing vars from separate files: https://promptfoo.dev/docs/configuration/guide#import-vars-from-separate-files You can use a javascript or python file to dynamically return the
context
var