https://promptfoo.dev/ logo
Confused about named metrics
# ❓|questions
m
I see how named metrics work with something like
similarity
where they aggregate the cosine similarity. With an assert like
llm-rubric
though, it seems to always accumulate 0 or 1, regardless of the
weight
of that assert. Is there some way I can have a named metric that adds up weighted `llm-rubric`s? (If you are interested, my use case is a bot that evaluates sales call, and I want to score them against a rubric of what our senior sales leader would have commented on those calls. He's provided those with weights, e.g. "10 points for positive feedback that the rep had a positive tone", "3 points for negative feedback that the rep provided incorrect technical information". If you see a better way to do that, I'm all ears! My goal isn't so much pass/fail as to give a total score for my whole suite of evals, and then iterate on my prompts to get a higher score.) https://cdn.discordapp.com/attachments/1341570378031562813/1341572737948192879/image.png?ex=67b67c79&is=67b52af9&hm=585368eff8062402ab3f3f22ffb7394417145acfcd69a815f43524ef9f34e65e&
... or i'd be equally happy with any other way to display the sum or average of the weights of passing asserts, like the .92, .43 etc. below
i
cc @michaelmichaelmichael as he is pretty familiar with weighted metrics
m
@michaelnglowforge This might solve it for you: https://github.com/promptfoo/promptfoo/pull/3142 Should be in the next release
m
Ah smart but not, I don't think it will solve it because the custom scoring function doesn't receive the assert weights, so I can't calculate what I need. Also, I'd like to be able to compute a final score for a provider across all of its tests.
m
I'll see what I can do for you later today
m
Thank you!!