https://www.boundaryml.com logo
Powered by
🟡 Memory requirements for multiple baml clients i...
# questions
s
Hello! Would importing say 10 baml clients into a single django project require a large amount of memory?
b
Baml allows creating multiple files and sharing types/clients/functions across them!
Everything exists in a flat global namespace
But if you do want to put multiple baml clients in a single project… I don’t think there’s that much overhead? There’s definitely per-baml-client overhead, but it’s mostly proportional to the size of the baml_src
s
Great - thank you! Was just wondering since importing the baml client (from shared.prompts.baml_client import b) seems to be causing our system to run out of memory somehow after deploy. Will report back if I find any bugs or anything.
b
Hmm… I will say that I don’t actually know how much memory overhead there is per client
But each baml client should just be doing “import baml_py” and that shouldn’t really have any overhead to it
s
An update - I think we've figured out the root cause -> When gunicorn starts with preload_app = True, Django loads all modules - including our 23 baml_client packages - each of which calls BamlRuntime.from_files() and initializes its own Tokio async runtime, and after gunicorn forks the worker processes, the pre-initialized Tokio runtimes (with their thread pools, epoll file descriptors, and mutexes) become corrupted, causing the workers to hang due to the invalid internal state. @henry_75477 for visibility
b
Whoa, nice find!
This isn't the first bug we've seen with process forking, we'll try to look at this next week
@Vaibhav (BAML 🐑) is there anything we did for the multiprocessing bugs/workarounds that would be cross-applicable here?
v
possible that we can use threading instead of multiprocessing library with gunicorn
my hunch says this should be pretty easily solveable with a min repro. just need to spin up a django app, and the improve the way we construct the singleton.
s
Sounds good - thank you for the quick responses!
I think this is a forking issue in general - just found another location in our codebase that causes hanging while baml is used:
Copy code
def start_worker(worker_cls: Type[BaseKafkaWorker]):
    config = worker_cls.config()

    def start_group():
        threads = []
        for _ in range(config.num_threads_per_process):
            worker = worker_cls()
            t = DBConnectionSafeThread(target=worker.start)
            threads.append(t)

        for t in threads:
            t.start()
        for t in threads:
            t.join()

    processes = []
    for _ in range(config.num_processes):
        p = Process(target=start_group) # <-- This is forking it, causing a Segmentation fault
        processes.append(p)

    for p in processes:
        p.start()
    for p in processes:
        p.join()
This error goes away when we change the above code to:
Copy code
def start_worker(worker_cls: Type[BaseKafkaWorker]):
    config = worker_cls.config()

    def start_group():
        threads = []
        for _ in range(config.num_threads_per_process):
            worker = worker_cls()
            t = DBConnectionSafeThread(target=worker.start)
            threads.append(t)

        for t in threads:
            t.start()
        for t in threads:
            t.join()

    # TODO [ENG-xxxx]: Re-enable multiprocessing after fixing fork-after-init issues
    # with BAML client. For now, always run directly without forking.
    start_group()
    return

    # Original multiprocessing code (disabled for now):
    # processes = []
    # for _ in range(config.num_processes):
    #     p = Process(target=start_group)
    #     processes.append(p)
    #
    # for p in processes:
    #     p.start()
    # for p in processes:
    #     p.join()
Hopefully this is helpful for easier re-creation @baml-sam @Vaibhav (BAML 🐑) Just to state again for completeness - we have 23 different BAML clients in our codebase, not just 1 in the root dir
v
Ah got it. Thanks for this! I’m guessing it probably happens with base case 2 baml_clients as well. This is super thorough and thank you for this. This is gonna help us fix this issue or at least identify it with better errors.
PreviousNext
Powered by