Hi @jimmy_soup , thanks for reaching out and providing those details! To help us investigate further, could you please let me know the version of Promptfoo you're using? Additionally, are you using any model-graded metrics in your setup?
It would also be helpful to know if the issue started after any recent changes to your config file or environment. If you’ve made any modifications recently or noticed any other details, feel free to share those as well.