anthonyivn
04/13/2024, 3:21 PMtests and asserts in the evaluate() method, and also because of the fact that the model-based metrics are accessible through the assertions process instead of as standalone metrics that can be called without defining the assertions.ian
04/13/2024, 3:44 PManthonyivn
04/13/2024, 3:45 PMian
04/13/2024, 3:46 PManthonyivn
04/13/2024, 3:46 PMian
04/13/2024, 3:47 PManthonyivn
04/13/2024, 3:47 PManthonyivn
04/13/2024, 3:48 PMResultian
04/13/2024, 3:48 PMian
04/13/2024, 3:48 PManthonyivn
04/13/2024, 3:48 PMian
04/13/2024, 3:50 PManthonyivn
04/13/2024, 3:50 PMian
04/13/2024, 3:50 PManthonyivn
04/13/2024, 3:50 PManthonyivn
04/13/2024, 3:50 PMian
04/13/2024, 3:50 PManthonyivn
04/13/2024, 3:50 PManthonyivn
04/13/2024, 3:50 PManthonyivn
04/13/2024, 3:52 PManthonyivn
04/13/2024, 3:54 PMian
04/13/2024, 3:56 PManthonyivn
04/13/2024, 4:32 PManthonyivn
04/13/2024, 4:33 PMian
04/13/2024, 4:33 PManthonyivn
04/13/2024, 4:33 PManthonyivn
04/13/2024, 4:33 PMian
04/13/2024, 4:33 PMian
04/13/2024, 4:33 PManthonyivn
04/13/2024, 4:34 PManthonyivn
04/13/2024, 4:34 PMian
04/13/2024, 4:34 PMian
04/13/2024, 8:20 PMevaluateMetrics to the node library. Please have a look at the example when you have a min and let me know if that would fit your needs https://github.com/promptfoo/promptfoo/pull/669anthonyivn
04/14/2024, 5:32 AManthonyivn
04/14/2024, 5:38 AManthonyivn
04/14/2024, 5:53 AMyeah, post-hoc evals is a CLI feature I'd like to add to the package can you explain more what do you mean here?anthonyivn
04/14/2024, 5:53 AManthonyivn
04/14/2024, 8:27 AMian
04/14/2024, 12:11 PMvarsanthonyivn
04/14/2024, 12:11 PMconst checkResult = await runAssertions({
prompt: renderedPrompt,
provider,
test,
output: processedResponse.output,
latencyMs: response.cached ? undefined : latencyMs,
logProbs: response.logProbs,
cost: processedResponse.cost,
});
I notice that the runAssertions function code above has a output arg that is based on processedResponse.output
I think a more graceful method for handling the post-hoc evaluation method is to:
1. Allow user to define output field within the tests (under vars as an element or as a separate value outside of vars but still nested within the same tests var)
2. Check for every every test whether output field exists or not, and if they do exist then we can assume that its a post-hoc evaluation
3. If 2) is the case, then we can skip running the prompt and went straight to runAssertions with output arg using the output provided by the useranthonyivn
04/14/2024, 12:14 PMtestSuite structure for evaluate() to something like this
const results = await promptfoo.evaluate(
{
prompts: ["First prompt with {{question}} and {{answer}}', "..."],
providers: ["openai:gpt-3.5-turbo"],
tests: [
{
vars: {
question: "...",
answer: "...",
},
output: "..."
},
{
vars: {
question: "...",
answer: "...",
},
output: "..."
},
],
writeLatestResults: true, // write results to disk so they can be viewed in web viewer
},
{
maxConcurrency: 2,
},
);anthonyivn
04/14/2024, 12:55 PManthonyivn
04/14/2024, 1:01 PMian
04/14/2024, 1:22 PManthonyivn
04/15/2024, 2:51 AMian
04/15/2024, 3:01 AManthonyivn
04/15/2024, 3:03 AMcallApi since the outputs to be evaluated are already available, and (imo) gives the flexibility of the current evaluate method without the need to maintain a separate function interface from evaluate()anthonyivn
04/15/2024, 3:05 AMoutputs x asserts pattern is definitely useful, however it's a bit of a departure from the current design pattern and data structure used in evaluate() and the CLI command, so I am not sure if that is the best thing to do hereanthonyivn
04/15/2024, 3:31 AManthonyivn
04/15/2024, 12:59 PMian
04/15/2024, 1:01 PMoutputs x asserts pattern because I've heard the request several times before actually. But I'm not opposed to also adding an easy way to override the provider output. WDYT?
My only nit would be to rename to providerOutput to be a little more explicit. Because confusingly, output is actually an input to the assert :panthonyivn
04/15/2024, 1:05 PMvars and just define the output... 🤔
I can make the name change to providerOutputian
04/15/2024, 1:07 PManthonyivn
04/15/2024, 1:07 PManthonyivn
04/15/2024, 1:08 PMian
04/15/2024, 1:09 PMian
04/15/2024, 1:10 PManthonyivn
04/15/2024, 1:13 PMian
04/15/2024, 1:16 PMian
04/15/2024, 1:16 PMian
04/15/2024, 1:16 PMian
04/15/2024, 1:16 PManthonyivn
04/15/2024, 1:16 PManthonyivn
04/15/2024, 1:17 PMproviderOutput field from the filesanthonyivn
04/15/2024, 1:17 PManthonyivn
04/15/2024, 1:18 PManthonyivn
04/15/2024, 1:18 PMian
04/15/2024, 1:18 PMian
04/15/2024, 1:22 PManthonyivn
04/15/2024, 1:22 PManthonyivn
04/15/2024, 1:22 PMian
04/15/2024, 1:22 PManthonyivn
04/15/2024, 1:22 PManthonyivn
04/15/2024, 1:22 PMian
04/15/2024, 1:22 PMian
04/15/2024, 1:23 PManthonyivn
04/15/2024, 1:23 PManthonyivn
04/15/2024, 1:23 PM