https://promptfoo.dev/ logo
Promptfoo eval with temperature 0 generates differ...
# ❓|questions
d
This is quite a big issue, because is hampering trust in the tool. I have a number of promptfoo tests in a github repo. The same code checked out by different users and executed with
promptfoo eval
generates different failing test cases. Something else that I have noticed is that the exact same prompt generates different responses from the actual app. I should be using the same prompts and settings (temperature of 0) and yet the app every now and then is giving slightly different responses from propmptfoo. Any idea on how i can debug the issue or I can check what's going on?
k
Zeroing out the temperature is not enough to make most(?) models deterministic. For one, you get a different seed each time unless you specify one. But even then, there are issues: https://152334h.github.io/blog/non-determinism-in-gpt-4/
i
if you are using openai, try adding a
seed
param
d
thanks for the answers!
I am still facing the issue even with seed param unfortuantely, can anyone suggest a solution for this?
k
Even with temp of 0 and a static seed, models are not fully deterministic. Did you see my link above?
p
🤣
Just save the config with tests you want to run in a static way.
d
lol is it more of openai issue im suspecting
d
yes I have seen it, so there is no way to have this deterministic?
k
Not 100%, no. Some older and smaller models that aren't MoE maybe, but in general I suggest letting the dream of pure determinism go.