I''d like to run an llm rubric, but not fail the entire test if this single assertion fails. This is what I curently have:

job_id,clause_number,text,legislation_id,expected_compliant,expected_analysis_rubric,__expected1,__expected2,__expected3
job123,2,"Customers have 3 days to return defective products",consumer_law,FALSE,The analysis must state that the minimum is 7 days.,javascript: JSON.parse(output).Evaluation.Status === 'OK',javascript: JSON.parse(output).Evaluation.Compliant === (context.vars.expected_compliant === 'TRUE'),llm-rubric: The analysis should satisfy the rubric: {{expected_analysis_rubric}}

But it's obviously failing when the

__expected3

fails... How can I still run the rubric, but disregard its actual score?

Try setting the

threshold

to

0

What is the syntax to set the threshold for the llm rubric on the csv? Using yaml it's sort of clear to me. Not so much with csv. I tried adding

,__metric,__threshold

columns, however this applies the threshold on the total score, not just for the llm rubric.

Applying threshold exclusively to llm rubric isn't supported right now but I'll open a PR to add it