Hi everyone, I'm wondering how are you all versioning prompts (or rather baml-clients I should say)? I was thinking of generating a new baml-client versioning for every different prompt, so my human evaluators can check the differences while testing different prompts on same input. My usecase is to try out different schemas instead of the prompt text itself. Anyone worked on something like this?

We don't have anything built in for this, and there are a lot of ways to go about it from application code. Is this the kind of testing you want humans to do before launching? Or does some of your testing happen later, based on prod data? Different versions of your function can both send data to boundary studio, so you can have two separate versions checked out; they'll be searchable by function name but differentiable by the different schemas. You could also use a single version of the code, and use dynamic types to tweak the schema. If you're modifying the text of the prompt, there are more options, because changing the schema means changing the interface to the prompt: If it's manual testing prior to release, I'd recommend simply making several separate functions, and either copy-pasting the prompt and doing small changes, or wrapping the prompt as a template_string and sharing it across the use sites. If you want something more dynamic, based on production data, then a nice pattern is to have one BAML function, and parameterize it by a variable that signals which version of the prompt to use. Then you can use the variable in A/B testing.

My particular usecase is about testing prior to release, but I'm also interested in how people generally try out different prompts specifically with baml. my process is to put the generated baml client in a lambda layer, and then I can use it anywhere in any lambda function. But I now realise that this is not a good implementation for testing out multiple baml clients (each having different prompts, schemas, schema-field descriptions variations). I did consider keeping this limited to a handful of variations and just copy-pasting the prompts with keeping the schema small, but then that leads to much more maintenance. I was wondering how are other people handling testing out prompts to other people who have more domain knowledge than me. I guess I like the idea of using dynamic types, so maybe I can try that out as well.

Ah it kind of sounds like you want a canary deployment. In this setup, you'd make a new git branch, change your prompts and types throughout the system however you like, and deploy the system to a canary environment, which can receive some subset of traffic (either you hit it manually, or you programatically forward a fraction of your users to the canary one). Then you could look at prod data in boundary studio and make the call to deploy canary more broadly.