Mosec Group

Hi just curious how is Rust involved in Mosec?

It seems like there is zero bindings of Rust codes with Python but there is the Rust binary built with setuptools in mosec/bin

Yeah. They communicate with each other through the Unix domain socket.

anyone knows how to deal with this kind of  request in a Preprocess(MsgpackMixin, Worker) class:
prediction = httpx.post(
    "http://127.0.0.1:8000/inference",
    content=msgpack.packb({"image": [dog_bytes,dog_bytes]}),
)

do you want to send one image or a list of images?

You can reply in this thread so others can get the full context here.

to enable dynamic batching, you should configure a `max_batch_size` on the `append_worker` method like `server.append_worker(StableDiffusion, num=1, max_batch_size=4, max_wait_time=10)`

if you just want to check whether it works or not, you can set the max_wait_time to an extreme large value

If there are multiple pieces of data within one request, and multiple requests are grouped into a batch, how does it know which result corresponds to which request when returning the results

Good question. You will need to return a list that has the same length as the batched input. Also, this list needs to maintain the same order as the input. Let's say the batched input is `[['a', 'a'], ['b', 'b', 'b']]`, then the output shoule be `['2a', '3b']`

"So, is my inference logic correct? Even though I haven't finished writing it yet, right?"

you need to split the `batch_label` into different groups according to the `each_len`

If my inference has two branches, and it depends on a request parameter to determine which branch to follow, how should I handle this situation?

different model,and different  post process

so far we don't support this, I'm working on the multiroute feature which will be the solution https://github.com/mosecorg/mosec/issues/355

for now, you might need to create two services

Sorry to bother you, may I ask if the init of Preprocess needs to pass parameters, What should I do at server.append()

append_worker can check the doc https://mosec.readthedocs.io/en/latest/reference/interface.html#mosec.server.Server.append_worker

msgpack mixin

no, my preprocess  and inference class like this, is this support batching ?
https://cdn.discordapp.com/attachments/988641138317594706/1126002420296712273/image.png
https://cdn.discordapp.com/attachments/988641138317594706/1126002420728729691/image.png

Hey <@553984377969967124> , thanks for fixing the ARM issue 🙂 I plan to take a closer look at the project and contribute to some of the features. What is the status of https://github.com/mosecorg/mosec/issues/395 ?

This one has no update for a long time. You can take it if you want to have a try. Feel free to ask questions.

How much is the perf benifit of separating the web server into Rust while keeping the transformation logics in Python?

I saw the code to communicate between Python <-> Rust introduces quite some complexity and you have to channel stuff via sockets

rust web layer is much faster than python wsgi/asgi

while python has to use multiprocessing, there will be IPC cost, whether it's python <-> python or rust <-> python

the IPC part is based on Unix domain socket, which is faster than most of the other IPC methods

intuition wise makes sense, any benchmark/reference for this? like how much faster

https://gist.github.com/kemingy/5de644c21e58f48cd706e9be5af5f872
this tested the flask + gunicorn v.s. mosec