This message was deleted.

Hi Scott, can you explain more about your usecase? I’m sure what are the requirements for you to integrate ESRGAN?

hey @Aaron Pham, thanks for getting back to me...basically i need to have multiple endpoints, one for stable diffusion generation, one for upscaling an image (using ESRGAN or whatever compatible model)...i guess also im making an assumption that bento can wrap and expose any machine learning model tho that may be naïve not knowing fully how they are all made myself

so essentially you can create two different runners. One for sd, another for image. A pseudo code would look something like:

sd_runner = bentoml.transformers.get("stable_diffusion:latest").to_runner()
gan_runner = bentoml.pytorch.get("esrgan").to_runner()

svc = bentoml.Service("service", runners = [sd_runner, gan_runner])

@svc.api(input=bentoml.io.Image(), output=bentoml.io.Image())
async def generate_image(input_data):
    return await gan_runner.async_run(input_data)

@svc.api(input=bentoml.io.Text(), output=bentoml.io.Image())
async def text_to_image(input_data):
    return await sd_runner.async_run(input_data)

You can read more about how you can define our service here

Ok thank you so much, this helps a lot. I read all the pages (including services and apis) but it wasnt quite clicking...havent done python in many years but program daily in other languages...

No worries. Glad I can help.

hey, if you dont mind, i only had 2 other lil things i was curious about that would open everything up for my needs... 1) do u think it would be possible to pass the image around on the service between the two models so i could generate the image, then upscale it, without doing an extra round trip w the image and hitting the upscale endpoint seperately? 2) is there any way to get the progress on the service side? idea here is to open socket to user and show % (unless there is some python way im not seeing on the services to do that)

1. Do you think it would be possible to pass the image around on the service between the two models so I could generate the image then upscale it?

You can do this perfectly by making a sequential graph

@svc.api(input=bentoml.io.Text(), output=bentoml.io.Image())
async def generate_image(input_text):
    image = await sd_runner.async_run(input_text)
    return await gan_runner.async_run(image)  # upscale

Is there any way to get the progress on the service side?

I mean here by “progress” you mean how long it takes to run the inference? You can use library such as tqdm to show progress of the inference task. https://github.com/tqdm/tqdm

you are a godsend..ty! will dive into researching and experimenting

Recently on main (which will be included in our next release), we introduce a

bentoml.metrics

will allow user to create Prometheus metrics to inside their service. This plays well with these sorts of metrics

h = bentoml.metrics.Histogram("inference_duration", "Duration of a given inference graph")

@svc.api(input=bentoml.io.Text(), output=bentoml.io.Image())
async def generate_image(input_text):
    start = time.perf_timer()
    image = await sd_runner.async_run(input_text)
    res = await gan_runner.async_run(image)  # upscale
    total = time.per_timer() - start
    h.observe(total)
    return res