https://promptfoo.dev/ logo
Add names to custom python assertions
# ❓|questions
a
is it possible for me to define a type/name/label value for custom python assertion result(s) in the output file? When I tried running the custom python assertions I made I notice i don't see any assertion label value for those custom assertions in the output .json file
i
do you mean the name of the
metric
, so it shows in the UI as its own separate score?
a
It's so that every python assertion score can have a name associated with them in the result object
I notice that when you use multiple python assertions there's no name field for the metrics
p
im running into this issue as well. here's a side by side comparison of an llm rubric assertion and a custom python assertion. as you can see,
type
and
value
are empty for the python assertion https://cdn.discordapp.com/attachments/1230006913086001233/1230583767953965226/Screenshot_2024-04-18_at_2.19.16_PM.png?ex=6633d971&is=66216471&hm=9fdbbeddfa25670a29276f311229ff1b871b57ad48d2b2e76d006c5237b2c558&
i
oooh got it
thanks. Will put this fix on the todo list
p
thanks! in addition to that, I noticed in the ui that it wasn't passing or failing my custom python assertion according to my
threshold
. here my threshold is set to 0.5 under
defaultTest
in my yaml but in the ui the assertion is marked as passed even though the score is 0. is
threshold
supported for custom python assertions? if so, am I using it correctly? https://cdn.discordapp.com/attachments/1230006913086001233/1230586767367737456/Screenshot_2024-04-18_at_2.30.48_PM.png?ex=6633dc3c&is=6621673c&hm=91aed922b156d39d9f36cf07dbb0eae30197d7b823da8eb4ca081769fb4316d8& https://cdn.discordapp.com/attachments/1230006913086001233/1230586767623585842/Screenshot_2024-04-18_at_2.31.00_PM.png?ex=6633dc3c&is=6621673c&hm=e3dfc11a44936f0402274ee26e63cb28617ae759f00e42b54a55719a304900a1&
i
here is a fix for the assert threshold https://github.com/promptfoo/promptfoo/pull/690
still working on the labeling. what do you think would make sense to show? at a minimum,
type: python
and
value: <filepath>
p
yeah that would work! feels like that's logically consistent given what's displayed for other test types
a
Yeah agree with @peanutshawn
i
fix for missing assertion
type
and
value
here: https://github.com/promptfoo/promptfoo/pull/691