Hi I have a question about adaptive batching, I'd like to understand how bentoml manages the mapping between requests and response. In the doc, it says the input and output of the batchable api should be a

List

, is there are any sort of indexing for identifying the output (e.g. which element in the list should go to which request as response). Or is it relies on the order of the list? Thank you!

the order will be handled by bentoml internally. you would expect to get the output in the same order as the input

I see, thanks. I have another question regarding to the doc. in the doc, it says:

When using a synchronous endpoint in one Service to call a batchable endpoint in another Service, it sends only one request at a time and waits for a response before sending the next. This is due to the default concurrency of 1 for synchronous endpoints. To enable concurrent requests and allow batching, set the threads=N parameter in the @bentoml.service decorator.

I'm wondering if the threads=N has any relationship with number_of_workers ? What would be a good value for N?

number of threads refers to the python threads, while number of workers are processes. threads => the concurrency of a sync endpoint