https://bentoml.com logo
Join Slack
Powered by
Can I get help on this :arrow_up: ? We want to dep...
# ask-for-help
e
Can I get help on this ⬆️ ? We want to deploy bento's on our kubernetes, but we some strict rules that the OpenAPI spec should adhere to - and I can't figure out how to override it. It was easier in the previous BentoML approach (without classes) but now I can't figure out how to do this
j
@Frost Ming do you have some insights on this?
f
do you mean to change the url of openapi spec
c
@Elior Cohen is this what you are looking for? https://github.com/bentoml/BentoML/pull/5176
e
@Frost Ming @Chaoyu Kind of, but not only I need to change components, title, description, version, severs paths etc. The case is that this is being deployed within an infra of a bigger API server, so it has some strict specifications about paths, errors, documentation etc. Before class based bentoml servers it was quite easy, I just inherited from Service class editted there the 
openapi_spec
method to my liking nad created the 
svc = ...
from that - but now that it is decorator based, I can't override, because the decorator overrides my own implementation
c
How does the new proposal in the PR look? @Elior Cohen?
Copy code
You can override various OpenAPI operation fields:

* ``description``: Detailed API endpoint description
* ``tags``: List of tags for grouping endpoints
* ``parameters``: Query, path, or header parameters
* ``requestBody``: Request body schema and examples
* ``responses``: Response schemas and examples
* ``summary``: Brief summary of the endpoint
* ``externalDocs``: External documentation references
* ``deprecated``: Mark endpoint as deprecated
* Any other valid OpenAPI operation field

Basic Example
-----------

.. code-block:: python
    @bentoml.api(
        openapi_overrides={
            "description": "Custom description for this endpoint",
            "tags": ["custom-tag"],
            "parameters": [
                {
                    "name": "version",
                    "in": "query",
                    "required": False,
                    "schema": {"type": "string"}
                }
            ]
        }
    )
    def predict(self, text: str) -> str:
        return self.model.predict(text)
f
i have another proposal, expose the service factory so one can subclass and customize it like how you does for legacy service class
e
I think @Frost Ming proposal is more robust as it will enable customizations of more parts. Moreover @Chaoyu in your suggestion, effectively I override the existing parameters - but for example in our implementation we don't overwrite but rather edit. So we generate the original OpenAPI spec, and then we might enrich some text and stuff like that on top of the exisitng one
👍 1
c
Thanks for the feedback!
e
@Chaoyu @Frost Ming is there any temporary workaround? It's really critical in our setup, deloyments that their openapi.json does not adhere to the organization standard will not be accepted
f
a possible workaround is to mount an asgi app at the root path and overwrite /docs.json route
e
Can you maybe jot up a quick code exmaple?
f
Copy code
import bentoml
from fastapi import FastAPI
app = FastAPI()

@bentoml.service
@bentoml.asgi_app(app)
class MyService:
    @app.get("/docs.json")
    async def get_openapi_schema(self):
        return generate_openapi_schema()
Sure
e
Thanks I'll try it, much appreciated!
@Frost Ming hi, I tried this solution - where 
generate_openapi_schema()
in your code example returns a 
bentoml._internal.service.openapi.specification:OpenAPISpecification
instance. It fails, when I go to 
localhost:3000/docs.json
I get an internal server error and the process emits this log line {"level": "INFO", "message": "127.0.0.1:59938 (scheme=http,method=GET,path=/docs.json,type=,length=) (status=500,type=text/plain; charset=utf-8,length=21) 0.829ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-04T075042.199006Z"} And when I go to 
localhost:3000
I see only one endpoint, which is the 
/docs.json
endpoint as shown in the attached image. My implementation is just like you stated
Copy code
@bentoml.service
@bentoml.asgi_app(app)
class FCModel:
    bentomodel = bentoml.models.get(....)

    #... Some app code


    @app.get("/docs.json")
    async def get_openapi_schema(self):
        return get_openapi_spec(self)
where the 
get_openapi_spec
implementation is as follows
Copy code
def get_openapi_spec(service) -> OpenAPISpecification:
    og_spec = generate_spec(service)
    components = gen_custom_components(service.apis)
    paths = gen_custom_paths(service.apis, service.name)
    custom_spec = OpenAPISpecification(
        openapi=og_spec.openapi,
        info=Info(
            title=str(service.tag),
            description="bla bla bla",
            version="2.0.0",
        ),
        components=components,
        servers=og_spec.servers,
        paths=paths,
    )
    return custom_spec
I'd appreciate any help getting this to work
@Frost Ming?
f
Sorry, you need to return a JSON response like:
Copy code
return JSONResponse(get_openapi_spec(self).asdict())
> And when I go to 
localhost:3000
I see only one endpoint, which is the 
/docs.json
endpoint as shown in the attached image. But in the screenshot you are visiting 
/docs
which is the FastAPI endpoint for openapi spec
e
Thanks, I'm going to try that and report
Hi @Frost Ming I did what you said. Still doesnt' work unfortunately, I'm going to put here all the information relevant. First of all, this is my code
Copy code
@bentoml.asgi_app(app)
@bentoml.service
class FCModel:
    bentomodel = bentoml.models.get(MODEL_TAG)

    def __init__(self) -> None:
        # Load files

    @bentoml.api(input_spec=APIInput, output_spec=APIOutput)  # type: ignore
    async def predict(self, **params: t.Any) -> APIOutput:
        try:
            <http://logger.info|logger.info>("Predicting...")
            self.model.logger.debug(f"Input: {params}")
            output = self.model.predict(**params)
            out = APIOutput(
                cph=output["cph"], cph_par=output["cph_par"], model=output["model"]
            )
            return out
        except Exception as e:
            logger.exception(f"Caught exception: {e}")
            raise e

    @app.get("/docs.json")
    async def get_openapi_schema(self):
        return JSONResponse(get_openapi_spec(self).asdict())
When I go to localhost:3000 I get 
{"detail":"Not Found"}
. If I go to 
/docs.json
I get Internal Server Error If I go to 
/docs
I see the image attached Lastly, here are the log outputs of my service during these checks
Copy code
poetry run bentoml serve --development --debug service:FCModel
2025-02-12T09:15:50+0200 [WARNING] [cli] Converting 'FCModel' to lowercase: 'fcmodel'.
{"level": "WARNING", "message": "`bentoml.models.get()` as the class attribute is not recommended because it requires the model to exist at import time. Use `bentomodel = BentoModel('fuel_model:rrhjnnwdngc5e7a6')` instead.", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:15:51.445264Z"}
{"level": "WARNING", "message": "Converting 'FCModel' to lowercase: 'fcmodel'.", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:15:51.487616Z"}
2025-02-12T09:15:51+0200 [INFO] [cli] Installing handle_callback_exception to loop
2025-02-12T09:15:51+0200 [INFO] [cli] Registering signals...
2025-02-12T09:15:52+0200 [INFO] [cli] Starting master on pid 15892
2025-02-12T09:15:52+0200 [DEBUG] [cli] Socket bound at 0.0.0.0:3000 - fd: 3
2025-02-12T09:15:52+0200 [INFO] [cli] sockets started
2025-02-12T09:15:52+0200 [DEBUG] [cli] Initializing watchers
2025-02-12T09:15:52+0200 [DEBUG] [cli] cmd: /home/eliorc/Dev/projects/ml_operations/.venv/bin/python
2025-02-12T09:15:52+0200 [DEBUG] [cli] args: ['-m', '_bentoml_impl.worker.service', 'service:FCModel', '--fd', '$(circus.sockets._bento_api_server)', '--service-name', 'FCModel', '--backlog', '2048', '--worker-id', '$(CIRCUS.WID)', '--ssl-version', '17', '--ssl-ciphers', 'TLSv1', '--development-mode']
2025-02-12T09:15:52+0200 [DEBUG] [cli] process args: ['/home/eliorc/Dev/projects/ml_operations/.venv/bin/python', '-m', '_bentoml_impl.worker.service', 'service:FCModel', '--fd', '3', '--service-name', 'FCModel', '--backlog', '2048', '--worker-id', '1', '--ssl-version', '17', '--ssl-ciphers', 'TLSv1', '--development-mode']
2025-02-12T09:15:52+0200 [DEBUG] [cli] running service process [pid 15931]
2025-02-12T09:15:52+0200 [INFO] [cli] Arbiter now waiting for commands
2025-02-12T09:15:52+0200 [INFO] [cli] service started
{"level": "INFO", "message": "Starting production HTTP BentoServer from \"service:FCModel\" listening on <http://localhost:3000> (Press CTRL+C to quit)", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:15:52.058431Z"}
{"level": "WARNING", "message": "`bentoml.models.get()` as the class attribute is not recommended because it requires the model to exist at import time. Use `bentomodel = BentoModel('fuel_model:rrhjnnwdngc5e7a6')` instead.", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:15:52.989784Z"}
/home/eliorc/Dev/projects/ml_operations/.venv/lib/python3.10/site-packages/sklearn/base.py:380: InconsistentVersionWarning: Trying to unpickle estimator LinearRegression from version 1.6.0 when using version 1.6.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
<https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations>
  warnings.warn(
/home/eliorc/Dev/projects/ml_operations/.venv/lib/python3.10/site-packages/sklearn/base.py:380: InconsistentVersionWarning: Trying to unpickle estimator LinearRegression from version 1.6.0 when using version 1.6.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
<https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations>
  warnings.warn(
2025-02-12 09:15:53,654 | DEBUG | service | Inferer initialized with 124 models.
2025-02-12T09:15:53+0200 [DEBUG] [entry_service:FCModel:1] Inferer initialized with 124 models.
{"level": "INFO", "message": "Service FCModel initialized", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:15:53.654889Z"}
2025-02-12T09:15:53+0200 [DEBUG] [entry_service:FCModel:1] executing <function connect.<locals>.connector at 0x7399606fadd0>
2025-02-12T09:15:53+0200 [DEBUG] [entry_service:FCModel:1] operation <function connect.<locals>.connector at 0x7399606fadd0> completed
2025-02-12T09:15:53+0200 [DEBUG] [entry_service:FCModel:1] executing functools.partial(<built-in method execute of sqlite3.Connection object at 0x7399606aee40>, 'CREATE TABLE IF NOT EXISTS result (\n    task_id TEXT PRIMARY KEY,\n    name TEXT,\n    input BLOB,\n    status TEXT,\n    result BLOB,\n    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n    executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n    completed_at TIMESTAMP DEFAULT NULL\n)\n', [])
2025-02-12T09:15:53+0200 [DEBUG] [entry_service:FCModel:1] operation functools.partial(<built-in method execute of sqlite3.Connection object at 0x7399606aee40>, 'CREATE TABLE IF NOT EXISTS result (\n    task_id TEXT PRIMARY KEY,\n    name TEXT,\n    input BLOB,\n    status TEXT,\n    result BLOB,\n    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n    executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n    completed_at TIMESTAMP DEFAULT NULL\n)\n', []) completed
2025-02-12T09:15:53+0200 [DEBUG] [entry_service:FCModel:1] executing functools.partial(<built-in method commit of sqlite3.Connection object at 0x7399606aee40>)
2025-02-12T09:15:53+0200 [DEBUG] [entry_service:FCModel:1] operation functools.partial(<built-in method commit of sqlite3.Connection object at 0x7399606aee40>) completed
2025-02-12T09:16:02+0200 [INFO] [entry_service:FCModel:1] 127.0.0.1:33232 - "GET / HTTP/1.1" 404 (trace=94abef0fead716ffeab742dae8df5014,span=e4a4029510f5c19f,sampled=0,service.name=FCModel)
{"level": "INFO", "message": "127.0.0.1:33232 (scheme=http,method=GET,path=/,type=,length=) (status=404,type=application/json,length=22) 0.714ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:16:02.486370Z"}
2025-02-12T09:16:28+0200 [INFO] [entry_service:FCModel:1] 127.0.0.1:33248 - "GET /docs HTTP/1.1" 200 (trace=530714f55ce6a20c91f314a1e5a8027a,span=63caf97f994b2c0d,sampled=0,service.name=FCModel)
{"level": "INFO", "message": "127.0.0.1:33248 (scheme=http,method=GET,path=/docs,type=,length=) (status=200,type=text/html; charset=utf-8,length=931) 0.694ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:16:28.350708Z"}
2025-02-12T09:16:28+0200 [INFO] [entry_service:FCModel:1] 127.0.0.1:33248 - "GET /openapi.json HTTP/1.1" 200 (trace=5379098a715d5123c1c832ac11578677,span=2a417568892d48d3,sampled=0,service.name=FCModel)
{"level": "INFO", "message": "127.0.0.1:33248 (scheme=http,method=GET,path=/openapi.json,type=,length=) (status=200,type=application/json,length=280) 1.010ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:16:28.718431Z"}
2025-02-12T09:16:30+0200 [INFO] [entry_service:FCModel:1] 127.0.0.1:33248 - "GET /docs.json HTTP/1.1" 500 (trace=c4691ec4826576f8e33084988d37e839,span=ee98fe7bd449dcd3,sampled=0,service.name=FCModel)
{"level": "INFO", "message": "127.0.0.1:33248 (scheme=http,method=GET,path=/docs.json,type=,length=) (status=500,type=text/plain; charset=utf-8,length=21) 8.240ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:16:30.798310Z"}
2025-02-12T09:16:33+0200 [INFO] [entry_service:FCModel:1] 127.0.0.1:49072 - "GET /docs HTTP/1.1" 200 (trace=bbfef99dfdac934b0052d3e6008bf00e,span=dea9586c6b6a4fbb,sampled=0,service.name=FCModel)
{"level": "INFO", "message": "127.0.0.1:49072 (scheme=http,method=GET,path=/docs,type=,length=) (status=200,type=text/html; charset=utf-8,length=931) 0.910ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:16:33.927203Z"}
2025-02-12T09:16:34+0200 [INFO] [entry_service:FCModel:1] 127.0.0.1:49072 - "GET /openapi.json HTTP/1.1" 200 (trace=a820f9e6730b15757bfbf5f7cf1ae7d5,span=07ccfadd11f7b499,sampled=0,service.name=FCModel)
{"level": "INFO", "message": "127.0.0.1:49072 (scheme=http,method=GET,path=/openapi.json,type=,length=) (status=200,type=application/json,length=280) 0.534ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:16:34.289956Z"}
2025-02-12T09:17:03+0200 [INFO] [entry_service:FCModel:1] 127.0.0.1:34228 - "GET /docs HTTP/1.1" 200 (trace=394ec9807fda664973f1b747313d4688,span=2019d10f939d8478,sampled=0,service.name=FCModel)
{"level": "INFO", "message": "127.0.0.1:34228 (scheme=http,method=GET,path=/docs,type=,length=) (status=200,type=text/html; charset=utf-8,length=931) 0.543ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:17:03.805810Z"}
2025-02-12T09:17:04+0200 [INFO] [entry_service:FCModel:1] 127.0.0.1:34228 - "GET /openapi.json HTTP/1.1" 200 (trace=dc863e181da44e369e493405556ee919,span=0c717ce898fdf935,sampled=0,service.name=FCModel)
{"level": "INFO", "message": "127.0.0.1:34228 (scheme=http,method=GET,path=/openapi.json,type=,length=) (status=200,type=application/json,length=280) 0.562ms", "trace_msg": null, "message_id": null, "email_id": null, "correlation_id": null, "fuel_model_model_tag": "fuel_model:rrhjnnwdngc5e7a6", "bentoml_request_id": null, "request_id": null, "@timestamp": "2025-02-12T07:17:04.190825Z"}
Hope you can help me sort this out
For reference - if I remove the 
@bentoml.asgi_app(app)
decorator - everything works fine (obviously not the swagger editing I'm trying to achieve) but as soon as I add it, I lose 
localhost:3000
and get 
{"detail":"Not Found"}
I keep debugging this, looks like I can't mount the asgi app to 
/
- any other path works
f
Make sure you are on the latest bentoml
/docs
is fastapi endpoint and is totally irrelevant in this context. All you need to check is 
/docs.json
returns valid response since it is used by 
/
to render the swagger page, and paste any error logs if you encounter internal server error. other info are not helpful
e
@Frost Ming gotcha, on it In my 
get_openapi_spec
I need access to the 
bentoml.Service
methods, the ones that essentially wrap my 
FCModel
(my own implementation). I need access to them because my 
get_openapi_spec
relies on attributes like 
.mount_apps
and 
app.title
app.routes
etc. But when I call 
bentoml.get_current_service
I get an instance of 
FCModel
without the attributes and methods of a 
bentoml.Service
. Is there a way for me, specifically for the 
/docs.json
endpoint I'm implementing to access the 
bentoml.Service
attrs/methods?
f
the service is the class itself, use 
MyService.some_method()
e
I'm afraid that's not true, sending a vid to showcase that
For example, I know 
bentoml.Service
has a 
context
variable, also I know the wrapper has a 
.inner
variable - both not available for example... I also tried using 
bentoml.get_current_service
Screencast from 2025-02-12 11-09-27.webm
@Frost Ming
my version is 1.3.21 btw
f
not 
self
but 
FCModel
get_openapi_spec(FCModel)
e
okay let me try
f
also 
bentoml.Service
is a wrong class to check against
e
I saw that is what the wraper returns
f
it’s for pre-1.2 style services
e
okay
what about the fact hta I can't mount the asgi app to 
/
?
f
you can, using the latest version
Copy code
import bentoml
import fastapi

app = fastapi.FastAPI()


@bentoml.service
@bentoml.asgi_app(app)
class MyService:
    @bentoml.api
    def generate(self, input: str) -> str:
        return input

    @app.get("/hello")
    def hello(self):
        return "Hello, world!"
This simple service gives the correct index page
e
again you are right, it fails because something in my generation fails
@Frost Ming so I'm encountering a weird error, I'm using
Copy code
from bentoml._internal.service.openapi import make_api_path, generate_spec
and when I apply 
generate_spec
on my my 
FCModel
it fails. When I observe why it fails, inside 
generate_spec
there is a line that uses 
generate_service_components(svc)
(which is also a bentoml's function) and that function throws the exception AttributeError: 'APIMethod' object has no attribute 'input' on the line
Copy code
def generate_service_components(svc: Service) -> dict[str, t.Any]:
    components: dict[str, t.Any] = {}
    for api in svc.apis.values():
        api_components = {}
        input_components = api.input.openapi_components()  # <------- HERE
Why wouldn't 
generate_spec
work for a bentoml's service?
f
again you should import from 
_bentoml_sdk.service.openapi
e
I'm using a lot of the 
_internal
how can I tell which is needed to be imported from where? I tried simply replacing 
_internal
with 
_bentoml_sdk
but I get linting errors so I assume not all exists in the 
_bentoml_sdk
these are my imports for reference
Copy code
from _bentoml_sdk.service.openapi import generate_spec
from bentoml._internal.service.openapi import make_api_path
from bentoml._internal.service.openapi.specification import (
    Info,
    PathItem,
    Response,
    MediaType,
    Reference,
    Components,
)
from bentoml._internal.service.openapi.specification import OpenAPISpecification
from bentoml._internal.service.openapi.utils import REF_PREFIX
from bentoml.exceptions import InternalServerError
f
this should be fine
e
Okay some other parts of my code fail I think because of the input vs input_spec thing. Trying to figure it out now... I'll update
@Frost Ming I'm reliant on some code that looks like 
api.input.openapi_request_body()
and 
api.input.openapi_responses()
, where 
api
is an instance of 
APIMethod
. I see that 
input
has been replaced with 
input_spec
- but can you tell me how can I replicate the two methods I need, 
openapi_responses()
and 
openapi_request_body()
?
f
you can look at the code of generate_spec under bentoml_sdk package
e
💯
Thank you very much @Frost Ming seems like that is all, it works now
that wasn't easy at all 😅
Open in Slack
PreviousNext
Powered by