I am trying to replace DSPy with BAML in my existing project, but one of the difficulty I come across is that I have some pre-curated golden dataset defined as jsonl, and I can use DSPy's data loader to load them into Optimisation Samples with train test split. Is there such thing in BAML, so I don't need to maunally enter the Optimisation samples into test definition? I could have asked LLM to generate it or use some scripting to parse the dataset and convert them into BAML test, but again this will make it complicated to review the test cases. Ideally, data could have been stored seperately from the test logic.

there is currently not a way to load data into a test like this, you would need to programmatically construct the

test .... {

baml definitions unfortunately. But this kind of capability is on our radar

Thanks @aaronv (BAML) , good to know that the test can live seperately Another related test questions, currently I have a test with 3 assertion. When I run test, >=1 of it failed, but it doesn't say which one. Is there a way to see the which assertion failed, and what's the actual and expected output like how pytest would show? https://cdn.discordapp.com/attachments/1457903024302457058/1457908694762389625/image.png?ex=695db6c7&is=695c6547&hm=658b6cebe51e56a38fd1cc43873af2ef6eec9ccd89d1c9d463642b3fd89d1bf1&

yes you can add @assert({{ thename, expression }})

@aaronv (BAML) I am not sure what I got wrong, but I couldn't get it work. Tried with @assert instead of @@assert but also didn't work, can you help me out again? Context: We are mainly using langgraph for the orchestration, and used DSPy only to optimise the critical prompt of the program, there are a couple of reason I wanna try replacing dspy with BAML Reason: the only reason I used dspy is for prompt opt to extract structure output. BAML as advertised, is really good at structured output (I haven't tested myself fully, but playing around with it seems good), and it's more developer friendly as in knowing what's going into the LLM (I think this is the deciding factor for me). Now that BAML also supports prompt opt (less options than DSPy, need to test how it performs), if the optimistion can work as well, I don't see any reason why I would choose DSPy over BAML. Tho 1 thing worth notice, is that we have two critical LLM structured output extraction in our pipeline, and what we found was one was better off with GEPA, one was better off with MIPROv2. We found that GEPA doesn't like using exampler, while MIPROv2 has a heavy focus on using few shot examples. And our assumption is one of the task given it's complicated nature, could benefit more from learning from examples than complex instruction. Hence it might worth while for BAML to integrate more optimisation algo in the future. https://cdn.discordapp.com/attachments/1457903024302457058/1457924975976779971/image.png?ex=695dc5f1&is=695c7471&hm=09577a4f3b4b6f503d1825e94b6ea717185fe9b1d6ea58730ff47382e7bf9c2a&

This is a quick easy fix. The name of the assert goes outside the {{ It looks like this: @@asset(some_name, {{ your expression }})

Thanks!!

no quotes, just @@assert(length_matches, {{ ... }} )

got it, thanks @aaronv (BAML) !!