https://promptfoo.dev/ logo
Re-Run failing Evals
# ❓|questions
p
Is it possible to re-run the failed evals unless we see some good results? I mean if i want to mark test pass if avg is more than 75%
l
Are you trying to test outputs that are non-deterministic due to high temperature?
p
yes
Hi @ian is this feasible? If i run 100 tests and if 10 of them are flaky/failing and if i re-run them with some custom script (as re-run failing eval is not a feature in Promptfoo) the final report will display execution status of only 10 tests not all the 100. Re-run of the only failing test is not possible in subsequent execution as it will generate the report based on last execution.
@here
i
Right now the best way to do this is to use the
--repeat
argument.
promptfoo eval --repeat 10
will run each test 10 times, and you can evaluate the overall pass threshold according to your needs
p
got it thanks
Just one more thing @ian the test will be concurrent or one at a time?
@ian just need your thoughts on this please?
i
Hard to say with just the screenshot. Even if it is running concurrently, it's still going to count up from 1, 2, 3...
2 Views