https://promptfoo.dev/ logo
Is there a way to permanently save thumbs up?
# ❓|questions
p
This is really the only thing I need to use promptfoo as daily prompt engineering tool. Ability to save manual evaluation thumbs up and downs - now it is cleared wherever I load another tab/section/run
i
Thanks for the feedback - working on this and expect to have a PR shortly.
p
That is great! Amazing response time - can’t wait for the next release
i
@paluchnuggets this PR is released in 0.34.0
p
I just downloaded newest version but still my thumbs up and down are not saved => when I change the run or reload viewer those thumbs disappear
i
Are you running with
npx promptfoo@latest eval
? Can you confirm the local version with
npx promptfoo@latest --version
or
promptfoo --version
?
p
yes, but now I noticed this error
ok I think I confirm that this is the problem, I have ran the test with variables of much smaller size and it works
Is it possible to overcome it? We often work with large variables (a lot of text).
i
😮 I see. That must be a very large eval. I'm pretty sure I can raise that limit on the server side.
fix here:
p
Unfortunately it still does not work with the same error => I have 10 test examples each has on avg 1.5k tokens (completon prompt) + around 500-700 tokens in one var that goes into prompt + there are other columns with some let's say metadata giving another 500 tokens so there is around 2.5k-3k tokens per example * 10 examples
first two are when editing very small run (everything works as expected), third is my attempt to give thumb up in a run of size I described above
i
Did you build from source? The fix is not in a release yet. Alternatively, if you have an
eval-*.json
that you can share with me, I can verify the functionality on my side.
Just published 0.34.1 - please give that a try 🙂
p
I did not build from source 😅- thanks for the release, I will check it today.
ok, it works 🙂 Thank you very much for adding this feature in such sort period of time.
2 Views