I think there's a bug in bentoML triggered here.

func_input = inference_api.input.from_arrow(batch)
                 ^^^^^^^^^^^^^^^^^^^
AttributeError: 'APIMethod' object has no attribute 'input'

load_bento can return

bentoml._internal.service.service.Service | _bentoml_sdk.service.factory.Service[Any]

While

service.Service

returns APIs of type

InferenceAPI

which has

input

defined,

factory.Service

returns APIs of type

APIMethod

, which doesn't have this attribute.

I see, you are using BentoML < 1.2, is that right?

I'm using 1.3.10

do you have a reproducer code?

More or less. Currently, it will fail regardless until this fix is released https://github.com/bentoml/BentoML/pull/5052 But considering you're running a version of bentoML with that fix, you should be able to use any bento service that uses the new service structure (

bentoml.Service

(old) ->

@bentoml.service()

(new)) and, running in spark do something like:

import bentoml
import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    StructType,
    StructField,
    FloatType,
    LongType
)

bento = bentoml.import_bento('/dbfs/mnt/<path_to_you_bento>.bento')

spark = SparkSession.builder.getOrCreate()

schema = StructType([
    StructField("col1", FloatType(), True),
    StructField("col2", FloatType(), True),
    StructField("col3", FloatType(), True),
    StructField("col4", FloatType(), True)
])

output_schema = StructType([
    StructField("field1", LongType(), True)
])

data = [[5.9, 3.0, 5.1, 1.8],[5.9, 3.0, 5.1, 1.8],[5.9, 3.0, 5.1, 1.8],[5.9, 3.0, 5.1, 1.8]]

# Create a DataFrame
df = spark.createDataFrame(data, schema)
results = run_in_spark(bento, df, spark, output_schema=output_schema) 
display(results)

another question, is your bento build with the pre1.2 (old) service structure or the new one?

The bento was built with 1.3.8

yes, but with BentoML 1.3.8 you can still use the old service structure. Can you also share the code of your bento?

It was built with the new service structure

import bentoml
from typing import Annotated
from bentoml.validators import DataframeSchema
import time
import dill
import pandas as pd


@bentoml.service(
    resources={"cpu": "1"},
    traffic={"timeout": 10},
)
class MyClass:
    def __init__(self) -> None:
        # Load inference model
        (...)

    @bentoml.api
    def method(
        self,
        arg
    ) -> myReturn:
        (..)

        return (...)

cc: @Frost Ming

I believe the code wouldn't fail if I was using the old service structure, as that would return a

bentoml._internal.service.service.Service

service, which has an

input

attribute. However, using the new service structure, we get a

_bentoml_sdk.service.factory.Service[Any]

service, which doesn't have an

input

attribute, so

func_input = inference_api.input.from_arrow(batch)

fails here

You are right, the spark batch only supports legacy

bentoml.Service

code. and we haven’t upgrade it yet

Is there a timeline I can refer to? Or some measures that can be taken to improve compatibility? (besides only using legacy service-based bentos) This was not communicated, and doesn't seem very stable.

we’ll try to make this work in the next release

Great to hear, thank you for the help. I'll test the new release as soon as it's out.

Sorry for late reply, after out internal discussion, conclusion are made that we are not going to support spark batch run in new service, at least not in the old way. So we will not update the code to make it work on the new services

Thanks for the feedback!