BAML: Concurrency & API Flexibility Issues?
# helpv
vaishnavi_j_82499
08/27/2025, 2:32 PMI’ve been exploring BAML and noticed there can be some challenges around concurrency and API flexibility. Could anyone share more details on:
1.The specific concurrency issues BAML currently faces (e.g., handling parallel requests, queueing, or throughput bottlenecks).
2.Any limitations in API flexibility (for example, strict message formatting, limited streaming support, or difficulty integrating custom workflows).
If there are ongoing efforts or recommended workarounds to mitigate these issues.
b
baml-sam
08/28/2025, 1:26 AM1- We’ve never had a user report a concurrency bottleneck beyond what their hardware is physically capable of, and since our runtime is implemented in Rust rather than Python/TS there’s a lot of memory allocation overhead that we don’t have to deal with
baml-sam
08/28/2025, 1:30 AM2- We’ve historically focused on just abstracting out single LLM chat request/response calls, and the iteration cycle there (testing, observability, evals, guardrails), because that’s the most flexible and composable primitive. We’ve got a bunch of stuff in the pipeline, but I’d rather ask you: what limitations have you noticed that you’re concerned about? What’s been your experience trying it?
v
Vaibhav (BAML 🐑)
08/28/2025, 7:42 AMalso @vaishnavi_j_82499 if you are running into any performance issues, please do share them with us! those are some of the most high priority issuses for us and we do try our best to make sure there's no speed / quality issues when using BAML in any way
v
vaishnavi_j_82499
08/28/2025, 12:14 PM@baml-sam I’m currently working on building a BAML prompt library for my organization. As part of this, we’ll likely need to run multiple baml functions concurrently. Before we proceed, I wanted to check — are there any known concurrency issues with BAML when executing functions in parallel? Would this cause problems specific to BAML’s runtime, or should we expect it to handle concurrent executions reliably?
vaishnavi_j_82499
08/28/2025, 12:17 PMAlso, I’m working on a prompt engine with BAML where I need to send multimodal to an LLM. My plan was to convert the multimodal data(like image,video,audio,etc.) into Base64 in BAML and send to LLM.
But I’m not sure what the best practice is if the target LLM doesn’t support Base64 multimodal inputs. Is there any recommended approach in BAML for handling unsupported Base64 formats?
b
baml-sam
08/28/2025, 5:28 PMThe BAML runtime is designed for concurrency out of the box! I'm not aware of any open issues we have around concurrency right now.
I would suggest using
default_client_mode asyncbaml-sam
08/28/2025, 5:32 PMOn your second point, base64 works with every LLM provider that supports multimodal inputs. The API that isn't consistently exposed is passing in a URL string to an uploaded asset - in those cases the runtime will automatically download and b64-encode the asset.
v
vaishnavi_j_82499
08/29/2025, 3:40 PMThanks, will try it out👍
vaishnavi_j_82499
08/30/2025, 7:57 PMHi , I am getting runtime errors in async_client.py after changing default_client_mode to async in generator.baml.
Pylance shows: Expected 4 positional arguments for reportCallIssue()
Error occurs in the auto-generated baml_client/async_client.py file
Specifically in the line: result.cast_to(types, types, stream_types, False, __runtime__)
https://cdn.discordapp.com/attachments/1410270793182740548/1411439579306659992/image.png?ex=68b4a90a&is=68b3578a&hm=fcf6ea07c36a4383759e57fc202330a07cce495b8508b98a02542a3f7eb52dcd&
b
baml-sam
08/30/2025, 9:34 PMYikes - looks like we had a regression in there at some point. We switched to exposing
sync_clientbaml-sam
08/30/2025, 9:34 PMWill get this fixed asap
baml-sam
08/31/2025, 12:17 AMHmm. What version of baml are you on? I'm not seeing this error on 0.206.1, which is the latest one we've released
v
vaishnavi_j_82499
08/31/2025, 6:49 AMI am using version 0.206.1, both for extension and baml-py. Is there anything i am missing here?
https://cdn.discordapp.com/attachments/1410270793182740548/1411603859520098344/image.png?ex=68b5420a&is=68b3f08a&hm=087b0cf50a73178585b4180f90692f966a2ad2a4a09a338d79209e709902eeef&
b
baml-sam
08/31/2025, 7:47 PMOh, I see, there is an issue with the pyi annotations on our internal interfaces. I'm not seeing any issues with actually running any of the async code though - what's the runtime error you're actually seeing?
baml-sam
08/31/2025, 7:47 PMWe also don't recommend checking in the generated baml_client FYI, that should be gitignore'd.
baml-sam
08/31/2025, 8:00 PMhttps://github.com/BoundaryML/baml/pull/2400 going in to address the pylance issues
v
vaishnavi_j_82499
08/31/2025, 8:21 PMWill try it out! thanks