https://promptfoo.dev/ logo
Is promptfoo currently designed as a framework mai...
# ❓|questions
a
After testing out promptfoo today and trying to explore the different flow/mode of operation that it has today, I am wondering if promptfoo today is mean to be designed for doing LLM/RAG testing instead of evaluating the actual outputs of LLM systems that are deployed out there. The reason why I am thinking this way is mainly due to the prominance of
tests
and
asserts
in the evaluate() method, and also because of the fact that the model-based metrics are accessible through the assertions process instead of as standalone metrics that can be called without defining the assertions.
i
@anthonyivn what's your ideal workflow? Are you looking to just calculate metrics on the fly rather than heavier test suites?
a
I can evaluate a set of data through the model-based metrics used for assertions
i
Ok, so you want to provide an existing list of inputs/outputs and have those graded
a
yeah, so that's basically a post-hoc evaluation process/flow, which I would say is kinda different from testing
i
Got it
a
So far I have managed to do this by using evaluate() and building my own calculation function around the result
where I basically ask the LLM to do certain things for me first before I run the calculation function using the
Result
i
Yeah, we sorta have a way to do this through the cli path ("standalone assertions") but it's not exposed in the library.
I can look into making this easier to do this weekend
a
when we use assertions, does evaluate() return the value produced from the metrics used for assertion?
i
Yes, the raw metric is in the returned data structure, nested somewhere
a
I see, so technically we can use assertions
i
Yeah you'd just have to process the object a bit
a
just set the threshold to 0, and ignore the verdict
yeah
i
I can provide a code pointer once I'm back at my computer
a
that sounds ok to me tbh
that would be great, thank you so much!
But in any case, what do you think about the question I asked at the start of this thread?
to be clear I am still going ahead with using promptfoo, I am asking the above mainly to understand the direction(s) promptfoo is going towards in the longer run
i
Understood 🙂 I do want promptfoo to be an excellent test framework. It's probably time to do a pass on the node library and bring it up to speed with the CLI capabilities so it's better for standalone tests. Most users use the CLI, which is more e2e
a
how about for post-hoc evaluations?
I think one thing that you might want to consider is that the CLI sort of forces you to define the reference data locally
i
yeah, post-hoc evals is a CLI feature I'd like to add to the package
a
which works for testing for sure
but not for post-hoc evals with larger data sizes
i
yeah
ok, let me see what I can do this weekend
a
ok thank you so much!
this conversation has been helpful for me
i
of course, lmk if you have anything else to flag
PR here to add
evaluateMetrics
to the node library. Please have a look at the example when you have a min and let me know if that would fit your needs https://github.com/promptfoo/promptfoo/pull/669
a
Ok I will take a look at it
Hey I took a look at it just now, I am not sure if adding only the LLM's responses to vars would be able to enable promptfoo to evaluate answer-relevance, context-relevance, context-faithfulness
yeah, post-hoc evals is a CLI feature I'd like to add to the package
can you explain more what do you mean here?
Are you referring to the number of args available through the CLI compared to the node package?
how about allowing user to provide vars and response to it from an existing system in prod (data coming in from logs), which then would allow post-hoc evals of metrics that are model-based or for RAG applications such as answer-relevance, context-relevance, context-faithfulness
i
just pushed an update to the PR to support
vars
a
Hey I spent some time today reading the codebase, and I think there might be a more cleaner solution for this use case:
Copy code
const checkResult = await runAssertions({
          prompt: renderedPrompt,
          provider,
          test,
          output: processedResponse.output,
          latencyMs: response.cached ? undefined : latencyMs,
          logProbs: response.logProbs,
          cost: processedResponse.cost,
        });
I notice that the runAssertions function code above has a
output
arg that is based on
processedResponse.output
I think a more graceful method for handling the post-hoc evaluation method is to: 1. Allow user to define
output
field within the
tests
(under
vars
as an element or as a separate value outside of vars but still nested within the same tests var) 2. Check for every every test whether
output
field exists or not, and if they do exist then we can assume that its a
post-hoc
evaluation 3. If 2) is the case, then we can skip running the prompt and went straight to runAssertions with
output
arg using the output provided by the user
If we do the above, then we can change the
testSuite
structure for evaluate() to something like this
Copy code
const results = await promptfoo.evaluate(
  {
    prompts: ["First prompt with {{question}} and {{answer}}', "..."],
    providers: ["openai:gpt-3.5-turbo"],
    tests: [
      {
        vars: {
          question: "...",
          answer: "...",
        },
        output: "..."
      },
      {
        vars: {
          question: "...",
          answer: "...",
        },
        output: "..."
      },
    ],
    writeLatestResults: true, // write results to disk so they can be viewed in web viewer
  },
  {
    maxConcurrency: 2,
  },
);
@ian https://github.com/promptfoo/promptfoo/pull/671 I wrote this PR just now to show the above suggestion in the codebase
let me know if you like the idea above, I can add examples or changes to the documentations similar to the PR you shared above
i
Thanks, I will take a look today!
a
Hey let me know if you need any clarification or changes made to the PR
i
sorry didn't get to this today The PR makes sense. I notice one difference here is the
evaluateMetrics
approach does
outputs
x
asserts
whereas yours is simpler and just gives people a way to attach a single output to each test case. I assume just adding
output
is sufficient for your use case
a
Yeah it would be sufficient for doing post-hoc evaluations, saves compute by skipping
callApi
since the outputs to be evaluated are already available, and (imo) gives the flexibility of the current
evaluate
method without the need to maintain a separate function interface from evaluate()
I think that the
outputs x asserts
pattern is definitely useful, however it's a bit of a departure from the current design pattern and data structure used in
evaluate()
and the CLI command, so I am not sure if that is the best thing to do here
And technically the evaluate() function today already does outputs x asserts, just that the outputs are produced by callApi()
Hey let me know if there's changes that you need me to make to the PR
i
was thinking about this. I might prefer to have the more flexible
outputs x asserts
pattern because I've heard the request several times before actually. But I'm not opposed to also adding an easy way to override the provider output. WDYT? My only nit would be to rename to
providerOutput
to be a little more explicit. Because confusingly,
output
is actually an input to the assert :p
a
Would that be more flexible though? technically with my change you can also do something like that by not defining values in
vars
and just define the output... 🤔 I can make the name change to
providerOutput
i
The pattern that I see is that people tend to ask "what are the metrics that I want across all my outputs". So constructing a test case for each output is cumbersome, especially if those test cases are repetitive
a
isn't that what defaultTest is for?
Sorry I might be misunderstanding how it works, but I was assuming that you can define the assertions that you want to pass to all tests by defining it there
i
yeah that's accurate
let's move forward with yours 🙂
a
Ok I made the change
i
coool thanks
merged!
I'll cut it in a release today
with a few other changes
a
Thank you!
I think additionally I will make PRs to modify the file readers to support the
providerOutput
field from the files
because as it is right now you can't access it from a csv file
But that's for another day
Thank you again
i
sounds good!
btw, I am not sure if the guy who requested this is your colleague, but I have a proposal for adding composite/derived metrics https://github.com/promptfoo/promptfoo/pull/670
a
yeah he is
he is from another product team
i
got it
a
tbh I am not sure how to best handle that logic..
i get why he wants it though
i
are you guys using promptfoo elsewhere at shopify, or just trying out different things?
yeah
a
we are part of the two teams starting out using promptfoo
likely other teams will follow as well
7 Views