Hello,
I need some guidance on a thing I am trying to do.
Using OpenAI I am trying to compare the output of strategies (system prompts, user post-prompts...) that may or may not use some function calling i defined.
When I assert on a test case, it evaluates the final LLM output and i have no information if the function calling has been called or not + if it has been called, was it done with the right arguments.
Context: My use case is a search assistant that parses from human text input some parameters and calls a private API to return the results. There are multiple cases i want to assert and it s not that easy, if i am using promptfoo well, to :
- Assert if the parameters have been parsed correctly: My work-around was to do a separate promptfooconfig and custom python script for that
- if the scenario goes user message1, assistant response1, user message 2, assistant message2, how to assert on the assistant response 1 and how to get the function calling (if there was one) before the response 1 : Still thinking of how to do this correctly without getting things messy.
If there is anyone that had this use case before and would share how he managed to do that that would be very helpful ! (I will too share how i did it, contributing to add a "complexe case" scenario in the docs maybe ?)
Thanks in advance