https://promptfoo.dev/ logo
llm-rubric running twice instead of once
# ❓|questions
n
Hey, I'm possible doing something wrong here, but I haven't been able to figure out what. When using scenarios and a single llm-rubric eval, the eval seems to run twice. (The UI shows "2 pass" and so does the json output.) I've modified the scenario example on github to reproduce this. Is this a bug or a mistake on my part? https://github.com/aantn/promptfoo/blob/605e52391295b099259f003c8d905fe01852b270/examples/multiple-translations-scenarios/promptfooconfig.yaml https://cdn.discordapp.com/attachments/1254127733408137410/1254127733642891455/Screenshot_2024-06-22_at_20.34.34.png?ex=66785cf8&is=66770b78&hm=a34b289f7a91d9350d663f0a7df8cb7bfe8729ac80ee69e564c4c06a55be9e09&
s
i had this yesterday 🙂 if you click the magnifying glass on the result, and use the right-most scrollbar you should be able to check what assertions were run. i think it's related to using both scenarios and defaultTest -- i restructured my tests after that and i'm back to a single assertion.
n
I see the same assertion running twice
Yep, I think its a bug with scenarios
@ian would it be useful to open a github issue for this?
s
in my restructure, i'm still using scenarios but removed the defaultTest and i'm not getting the double assertions.
n
Ah, that could work for me
s
yes -- honestly, i've found that having simpler basic configs, a few folders and running different evals from the command line is working better for me than bigger combined fixtures so far (and it's easy enough to put specific cases in shell scripts).
3 Views