Hey everyone! Wanted to crowdsource some ideas here.
First, let me explain how I use promptfoo right now.
I usually start by creating a typescript promptfile and some mock tests until I get a prompt that feels good. Then I put it into my app and collect some data. I take that data and convert it into evals (tests.yaml). This is all great!
The problem is when I start to update my prompt over time. Maybe I change one or a few of the variables. Now my entire tests file is out of sync, and these old evals are getting stale.
Does anyone have a good solution they've implemented already?