https://promptfoo.dev/ logo
Define soft and hard test in a configuration
# ❓|questions
k
I have nearly 10-12 llm-rubric assertions for each test case. No matter what I do in the prompt it always fail to pass all the assertions. Is there a way to define some hard/soft assertions, where hard test must always pass and soft test if fail then also the we get the success on the overall test. The way I see it is, if I can get %success on assertions and if all hard tests pass then I see the overall test is passes. It will be good if I can get this output as a result in command prompt.
p
I am wondering this as well
i
There isn't a great way to do this currently, but definitely open to implementing. Any proposals for what a good config/API structure would look like? In theory you could set the [assertion weight](https://promptfoo.dev/docs/configuration/expected-outputs/#weighted-assertions) very low for assertions you don't care about, but that just affects the score and not the pass/fail UI