Hi team, has anyone run into an issue where all the endpoints are working as expected, but the Swagger UI on localhost:3000 fails to load with "Failed to load API definition" Below is the attached log of the app until the error. Any help will be much appreciated. Thank you!

2024-05-22T15:48:07+0000 [INFO] [cli] Applying user config override from ENV VAR: tracing.sample_rate=1.0
2024-05-22T15:48:07+0000 [DEBUG] [cli] GPU not detected. Unable to initialize pynvml lib.
2024-05-22T15:48:07+0000 [INFO] [cli] Installing handle_callback_exception to loop
2024-05-22T15:48:07+0000 [INFO] [cli] Registering signals...
2024-05-22T15:48:07+0000 [INFO] [cli] Starting master on pid 17
2024-05-22T15:48:07+0000 [DEBUG] [cli] Socket bound at 0.0.0.0:3000 - fd: 7
2024-05-22T15:48:07+0000 [INFO] [cli] sockets started
2024-05-22T15:48:07+0000 [DEBUG] [cli] Initializing watchers
2024-05-22T15:48:07+0000 [DEBUG] [cli] cmd: /usr/local/bin/python
2024-05-22T15:48:07+0000 [DEBUG] [cli] args: ['-m', '_bentoml_impl.worker.service', 'service:{service_name}', '--fd', '$(circus.sockets._bento_api_server)', '--service-name', '{service_name}', '--backlog', '2048', '--worker-id', '$(CIRCUS.WID)', '--prometheus-dir', '/home/bentoml/prometheus_multiproc_dir', '--ssl-version', '17', '--ssl-ciphers', 'TLSv1']
2024-05-22T15:48:07+0000 [DEBUG] [cli] process args: ['/usr/local/bin/python', '-m', '_bentoml_impl.worker.service', 'service:{service_name}', '--fd', '7', '--service-name', '{service_name}', '--backlog', '2048', '--worker-id', '1', '--prometheus-dir', '/home/bentoml/prometheus_multiproc_dir', '--ssl-version', '17', '--ssl-ciphers', 'TLSv1']
2024-05-22T15:48:07+0000 [DEBUG] [cli] running service process [pid 32]
2024-05-22T15:48:07+0000 [INFO] [cli] Arbiter now waiting for commands
2024-05-22T15:48:07+0000 [INFO] [cli] service started
2024-05-22T15:48:07+0000 [INFO] [cli] Starting production HTTP BentoServer from "service:{service_name}" listening on <http://localhost:3000> (Press CTRL+C to quit)
2024-05-22T15:48:09+0000 [INFO] [entry_service:{service_name}:1] Applying user config override from ENV VAR: tracing.sample_rate=1.0
2024-05-22T15:48:09+0000 [DEBUG] [entry_service:{service_name}:1] 'tracing.sample_rate' is set to zero. No traces will be collected. Please refer to <https://docs.bentoml.com/en/latest/guides/tracing.html> for more details.
2024-05-22T15:48:12+0000 [INFO] [entry_service:{service_name}:1] 192.168.65.1:20751 - "GET / HTTP/1.1" 200 (trace=21083ed611c6b761364e927c0621c3d0,span=f38ed4e98f2c0405,sampled=0,service.name={service_name})
2024-05-22T15:48:12+0000 [INFO] [entry_service:{service_name}:1] 192.168.65.1:20751 (scheme=http,method=GET,path=/,type=,length=) (status=200,type=text/html; charset=utf-8,length=2945) 5.450ms (trace=21083ed611c6b761364e927c0621c3d0,span=ba1909ce8444359d,sampled=0,service.name={service_name})
2024-05-22T15:48:12+0000 [ERROR] [entry_service:{service_name}:1] Exception on /docs.json [GET] (trace=5570f598cbded602e4492ead8b184900,span=2970c098242598df,sampled=0,service.name={service_name})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/_bentoml_impl/server/app.py", line 165, in openapi_spec_view
    return JSONResponse(self.service.openapi_spec.asdict())
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bentoml/_internal/service/openapi/specification.py", line 282, in asdict
    return bentoml_cattr.unstructure(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dd_tracer/python/cattrs/converters.py", line 238, in unstructure
    return self._unstructure_func.dispatch(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<cattrs generated unstructure bentoml._internal.service.openapi.specification.OpenAPISpecification>", line 11, in unstructure_OpenAPISpecification
    res['components'] = __c_unstr_components(instance.components)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dd_tracer/python/cattrs/converters.py", line 965, in unstructure_optional
    return None if val is None else _handler(val)
                                    ^^^^^^^^^^^^^
  File "<cattrs generated unstructure bentoml._internal.service.openapi.specification.Components>", line 3, in unstructure_Components
    'schemas': __c_unstr_schemas(instance.schemas),
                                  ^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'schemas'
2024-05-22T15:48:12+0000 [INFO] [entry_service:{service_name}:1] 192.168.65.1:20751 - "GET /docs.json HTTP/1.1" 500

Hello, when you run

bentoml serve

does the swagger UI works as expected? Can you share your Service code?

Hey @Jian Shen Yap Thank you for the help! Yes, the swagger UI works when I just run

bentoml serve

, so I think there is something with customizing the ENTRYPOINT that may be breaking it (?). I am customizing the ENTRYPOINT to integrate datadog into the service. I am deploying the service to Google Cloud Run (serverless). By service code, are you referring to service.py?

yes, your

service.py

and maybe also your

bentofile.yaml

I think the error maybe be due to the

bentoml

version within the docker container is different than your local environment but I can't be sure

bentofile.yaml,service.py

could you check the version of

bentoml

and

cattrs

in your local environment as well as the running container?

local environment

❯ bentoml --version
bentoml, version 1.2.9

❯ cattrs --version
zsh: command not found: cattrs

for

cattrs

can you run

pip freeze | grep cattrs

container

$ pip freeze | grep cattrs
cattrs==23.1.2

local

❯ pip freeze | grep cattrs
cattrs @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_b0b_jdcke7/croot/cattrs_1699984884884/work

if you dont inject the datadog integration, does the swagger UI works?

Yeah, if I run without using the Dockerfile.template, it works, which makes me think it's something with customizing the ENTRYPOINT Running with dockerfile.template commented out

service: "service:MLService"
name: mlservice
description: "file: ./README.md"
include:
exclude:
  - ".env"
  - "venv/"
  - ".gitignore"
  - ".gitattributes"
  - ".gitlab-ci.yml"
  - ".vscode/"
  - "__pycache__/"
  - "*.json"
  - "*.ipynb"
python:
  requirements_txt: "./requirements.txt"
# docker:
#   dockerfile_template: "./Dockerfile.template"

FWIW, I had this same issue with the same error when trying to startup with

ddtrace-run bentoml serve

. Never got to the bottom of it, since instead just adding the middleware and using the init injection container in a K8s deployment worked for us.

Thank you @Adam Liter. Would you know if the same would apply for our case where we are deploying to google cloud run as a serverless app instead of K8s deployment? Far from an expert in K8s, but my understanding is that with K8s, we are able to spin up a separate agent host container that monitors the app container. But not sure how to do that in our infra.

I’m not sure. We don’t use GCP, so I’m not familiar with Google cloud run. But I would imagine that as long as the environment variables are set correctly in the container/machine that’s running on Google cloud run, you probably don’t need either

ddtrace-run

or the

/app/datadog-init

. Maybe the DataDog people can help you figure out which environment variables need to be set in order for this to work with only adding the middleware to the service code? That being said, @Jian Shen Yap: the recommendation from DataDog for ASGI applications involves using ddtrace-run. The fact that this doesn’t work seems like an issue. I’m not sure if it’s a bug in

bentoml

,

ddtrace

, or both. The fundamental issue seems to be that

ddtrace-run

tries to do something with the OpenAPI spec that

bentoml

generates , and this just doesn’t work. I didn’t dig into it further, since I found something else that works for us at our organization.

@Adam Liter I created a support ticket with datadog and this is an alternative that they suggested.

I'd expect using OTel to work as an alternative, given BentoML comes with built-in support for OpenTelemetry. The added benefit there is that it's vendor-agnostic (in case you ever use other monitoring tools) and it doesn't require installing the Datadog Agent on a separate host, and instead uses the OTel Collector and Datadog Exporter to ingest spans/traces from your app into Datadog:

https://docs.datadoghq.com/opentelemetry/collector_exporter/

I'll try this out and let you know how it goes.