And forgot to mentioned, I really really appreciate that you incorporated this feature into DSPy. I used DSPy for my LLM pipeline, but it wasn't easy to setup, and BAML is really simple to get started with. Now having optimisation embedded, I don't need to bear with the complexity of DSPy anymore, so thanks for your work on this!

For a meeting, how about 9 or 10AM your time Friday or Tuesday? (if I've done the conversion right, that would be 2pm/3pm Thursday or Monday for me (I'm in Pacific time)

Hey Greg, thanks for the call, it was really helpful! I think I confused myself a little bit with the FN and FP thing at the end, as that only applies to classification task, just want to clairfy so you are not confused by me. In general, the jsonl that constructs the test set should contain only the positive/correct examples. Below is the actual use case we use dspy for in prod, which aims to parse the customer product query into a schema: {"query": "organic bananas 1kg", "entity": "bananas", "attributes": ["organic"], "brand": "", "pack_count": "", "pack_size": "", "pack_type": "", "measure_value": "1", "measure_unit": "kg", "specials": "NONE"}

query

is the "input",

the rest

are the "expected output" that the LLM should produces. We probably should do a nested structured with input and output field, but this is what we have currently.

So what we do is to have a bigger LLM as a judge in production to evaluate if the output parse from the smaller LLM is correct with random sampled (cost concern). If any of the value is not correct, the bigger LLM will suggest the correct values and append to our curated test set. Then after a while (haven't happened), we as developer should retrain the smaller LLM using GEPA and the augmented test set. The hope is that the bigger LLM can make the smaller LLM better overtime purely by prompt optimisation. Like the student-teacher finetuning for LLM, but only for prompt not weight. In an ideal world the re-train could be automated with some MLOps, but not happening yet. Either case having the testset specified as jsonl or other data format would be benefitial. So with this setup, all the jsonl data point would be loaded automatically into baml test case with something like

@@check this.body==exoected_output

. I am not sure if having custom check/assert condition supported by jsonl could add much benefit, probably more of a could-have feature, at least in my use cases so far. Hope this helps!

Also https://github.com/gepa-ai/gepa?tab=readme-ov-file#optimize_anything-beyond-prompts recently has evolved from prompt optimisation to optimise any text artifact (code, config, etc). This kind of tie to what I was asking about the possibility to expand into optimising the structured output signiture definition. Just FYI in case you are considering implementing that feature. They are framework agnostic as well, which means is possible to make it as an optional baml dependency and directly call it out of box for baml app.