https://bentoml.com logo
Join Slack
Powered by
Hi! Is there an easy way to run code after the Inf...
# ask-for-help
m
Hi! Is there an easy way to run code after the InferenceAPI has returned the response? (Similar to BackgroundTask from Starlette)
s
Yes, but depends one what task you’re looking to do. Could you please share some details?
m
I want to schedule pushing the predictions to a Kafka topic to happen in the background (after the response has been returned)
I see in the HTTPAppFactory that the Response is obtained from the IODescriptor. Could I create an extension of my IODescriptor to return an Starlette Reponse containing the BackgroundTask?
something like:
Copy code
class BackgroundJSON(io_descriptors.JSON):
....
async def to_http_response(..., ctx):
   response = super().to_http_response()

   ... # check ctx is present

   response.background = ctx.background
   return response
Sadly, I would need to change ServiceContext as well
I wish there was a way to access to Starlette Response object more easily
s
I wish there was a way to access to Starlette Response object more easily
Thanks for the feedback. We are evaluating designs on exposing the Starlette response.
For the time being, would it be acceptable to send the Kafka event in a separate thread (if function sync) or through asyncio (if function is async), right before the function returns?
m
but who would wait for that thread/async function?
It’s the last thing I do in the api (after model had run)
so launching something just to wait for it seems like no gain
s
There’ll be no wait. It’s going to be fire-and-forget.
Copy code
@bentoml.api
def predict(self, arg):
    # model inference
    result = self.model.predict(arg)

    def send_kafka_event():
        ...
    # Start a new thread for the task
    thread = threading.Thread(target=send_kafka_event)
    thread.start()

    return result
m
oh, right, the good old thread
s
If function is async, 
asyncio
has some more elegant functions.
m
yeah, I was checking 
anyio
as well
Ty Sean!
👍 1
s
create_task
is what you are looking for if using 
asyncio
.
v
I am pretty sure I tried asyncio.thread in this example and it still blocked....but I can try the Thread as well
or even this 
<http://anyio.to|anyio.to>_thread
s
@Vasileios Zografos Python GIL makes it tricky. Would you like to share a snippet maybe we can look together.
s
May I recommend building this module totally using async construct with Rust. Build a Wasm and totally get independent of using threads. It’s a typical waiting in parallel and should be done by “green” threads that you should have millions of. Just to add some contextual flavour. 🙃
2 Views
Open in Slack
PreviousNext
Powered by