https://promptfoo.dev/ logo
Is there an option to abort after x failures?
# ❓|questions
p
We would like to abort if an eval reaches more than 5 failures to not use credits unneccessary when a prompt is not working as intended. I imagine either a set number or a percentage of failures. There is no need for this to be exact (I imagine concurrency making it difficult to guarantee), evaluateOptions.maxFailures = { type: 'fixed', count: 5 } If asserts are done after running all the prompts, then I imagine this will be hard to do.
m
Hi @patricklef Please thank Liza for the great post. We do not support that but we could. Take a look at https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L804 and https://github.com/promptfoo/promptfoo/blob/main/src/envars.ts#L4 I would happily accept a PR if this is something you were willing to work on.
p
Nice, that looks simple enough. Will give it a try
m
Happy to pair with you later if you get stuck
u
I have a fork where I do a stop on fail flag set for the assertion and if any one of those fail return the assertions that were executed up until that point
If this is interesting I can put in a PR
m
Can you either PR it or share a link to your branch?
u
Sure I can PR it
2 Views