Hello Team, We have an API service built using

bentoml.Service()

. Due to the latency requirements and the size of our data, we've opted to load some datasets as global variables. However, one of these files is approximately 900MB. When the service is deployed, it spins up multiple workers, and each worker loads this file into memory independently. This has led to a significant increase in the container size, which is impacting our infrastructure costs. We are looking for advice on how to address this issue more efficiently. Specifically, we would like to explore solutions for sharing the file between multiple workers or any other optimizations that could reduce memory consumption and container size. We would greatly appreciate any insights or suggestions from the community on how to tackle this problem effectively. Looking forward to learning from your experiences. Best regards, Manoj Ravi

Hello! That's a valid concern. Have you tried memory-backed file

mmap

before? Theoretically the OS is smart enough to know that multiple process is reading the same file so it only load once, but we have not tested this before. Let me know if you would like to try!

Hello Jian, Thanks for your response !! I don't think the workers are loading only once and sharing it across because our server capacity has increased drastically. I have not tried mmap but how will it affect the internal processing of worker. Would be helpful if you could explain/guide for me to try.

instead of using

f.open

you use mmap which doesn't load the file into the actual memory but virtual memory which take less space.

with open("large_file.bin", "r+b") as f:
     mm = mmap.mmap(f.fileno(), 0)

The file is in parquest format, and loaded as dataframe.

Hello, sorry for the late reply. First you need to make sure that your file is a binary so that it can be memory mapped. You cannot work with a CSV.

numpy

has a

tofile

API to allow this. or you can use

py.arrow

Secondly, you would need to make sure the operations that you used work nicely with memory mapped data. For this I would suggest you to use numpy's operation rather than pandas, although pandas 2.0 said they have mmap file support but I am not completely sure on the internals. > file using memory map, and convert it to dataframe. This loading process is going to happen for all the workers right, Using memory mapped file, you are loading the data into Virtual Memory, not physical memory. It consumes significantly less memory. You can easily try this out by yourself. Lastly, if the operations requires you to load everything into the memory, i.e. barebone sorting, i'm afraid that you may not be able to get much improvement from

mmap

. You may want to consider rearchitecture your solution, such as making use of batching, or a database, etc..

Thanks for the detailed explanation @Jian Shen Yap !! Appreciate it !!🙏🙌