https://promptfoo.dev/ logo
Pass/Fail beyond weighted mean
# ❓|questions
b
Hi all, I have a case where I'd like to define a custom function to combine assertion scores. E.g. I'd like a test to fail if either (a) the weighted mean score is below the test threshold or (b) any one of the composite assert statements fails badly (e.g. score of 0.01), even if that assert has a low weight. From what I understand (drawn out in the attached image) the final test score is computed by a hierarchical weighted mean, which generally allows for some asserts to fail but the entire test to still pass. Is there a way to do define a custom score reduction formula, somewhat like a per-test derived metric? Thanks! https://cdn.discordapp.com/attachments/1296109878229602474/1296109878733045830/assertions.png?ex=671117e3&is=670fc663&hm=a0995960c0235dec780e025f1593a8afa2c3d626aefeda715d1683ac22eb1de8&
k
I'm interested in the answer to this, too. Or maybe even suggestions on best practices to handle these sorts of concerns (even if not exactly as requested above).
Any thoughts on this from the experts?
m
Hey @brorodin and @kentyman, I apologize for the delay. Thanks for your patience. It should definitely be possible to define a custom score reduction formula like what you’re describing (possibly with no code change). I’ll set aside some time tomorrow to look at this and provide a more detailed response. Thanks!
b
Hey, just checking in to see if you've got an update. I've been making due by post-processing the output json file with a python script but that's rather messy and not the most sustainable. Thanks!
k
Just checking in again on this, @michaelmichaelmichael
Friendly bump.
Happy New Year! Anything clarifications we can add to help this request along?
Friendly bump.
FYI @brorodin