michaelnglowforge
02/19/2025, 12:41 AMsimilarity where they aggregate the cosine similarity. With an assert like llm-rubric though, it seems to always accumulate 0 or 1, regardless of the weight of that assert. Is there some way I can have a named metric that adds up weighted `llm-rubric`s?
(If you are interested, my use case is a bot that evaluates sales call, and I want to score them against a rubric of what our senior sales leader would have commented on those calls. He's provided those with weights, e.g. "10 points for positive feedback that the rep had a positive tone", "3 points for negative feedback that the rep provided incorrect technical information". If you see a better way to do that, I'm all ears! My goal isn't so much pass/fail as to give a total score for my whole suite of evals, and then iterate on my prompts to get a higher score.)
https://cdn.discordapp.com/attachments/1341570378031562813/1341572737948192879/image.png?ex=67b67c79&is=67b52af9&hm=585368eff8062402ab3f3f22ffb7394417145acfcd69a815f43524ef9f34e65e&michaelnglowforge
02/19/2025, 12:51 AMian
02/19/2025, 6:39 PMmichaelmichaelmichael
02/19/2025, 6:40 PMmichaelnglowforge
02/19/2025, 7:03 PMmichaelmichaelmichael
02/19/2025, 7:04 PMmichaelnglowforge
02/19/2025, 8:06 PM