https://www.boundaryml.com logo
Powered by
Prompt Optimizer Issues
# help
k
I'm having a lot of trouble getting the prompt optimizer to do anything. It seems like it just runs through the tests and gives me a "complete!". It doesn't look like it's doing any actual updating of the prompt. I did change the gepa client to be Gemini 2.5 pro in vertex (my company can only use models in vertex). I've got a bunch of tests with assert statements, and I even intentionally gave a really bare bones prompt (accuracy only 34%) and it still didn't make any updates to it
g
Oh interesting. Thanks for the report! Could you share some screenshots when it’s complete? I wonder if this is a UI issue (it can be non-obvious how to accept a new optimized prompt). You can also run it with —no-tui to see more detailed output.
Are you able to show your BAML prompt and test cases? That may help me see what’s going wrong.
k
I can't send the prompt or test cases 🙁 it's all PII/PHI
what's interesting is that when I run the example of extracting the subject, it works great
I know not having the prompt or tests basically makes it impossible to debug so just wondering if you've seen this before
g
Huh, interesting. And your asserts are in the test block, not on the types themselves?
Hah, no, no worries. Having the exact prompt makes debugging too easy 🙂 It should be a bit of a challenge.
I haven't seen this exact thing, but I have seen optimization be finnicky in general, and we don't provide good error messages when something goes wrong.
I definitely want to help get you un-stuck. We can do it even w/out the exact prompt, if you don't mind some back-and-forth.
k
let me see if i can reproduce it with a complete dummy prompt and test cases
g
75 tests, btw. That's great!
k
ha I wrote a script to create them from a dataframe
g
that does give me an idea. I wonder if there's a bug in the optimization budget where it spends its whole budget on the first 1/10th of your tests.
Mm, can you try 
baml-cli optimize --from my_baml_project --max-evals 1000
?
The default is 50. This could be the issue 🙂
k
alright awesome I'll give it a try and let you know. gotta run for today though
THAT DID IT
🙌
g
Great! Ok so my task is to add some validations and helpful messages if the defaults would not work well for a given optimization run.
Thanks for the quick check!
a
oo cool, a nice msg about the status of the run or something will probably help
8 Views
PreviousNext
Powered by