https://bentoml.com logo
Join Slack
Powered by
This message was deleted.
# ask-for-help
s
This message was deleted.
🍱 1
a
can you send your service definition here?
e
Sure! It's fairly vanilla stuff:
The slow part I believe is line 11 or 13 when the model is actually loaded.
a
THe model is not loaded here, runner is simply a reference. How big is your model? can you send the output of 
bentoml env
?
e
Copy code
`bentoml`: 1.0.7
`python`: 3.10.4
`platform`: macOS-10.15.7-x86_64-i386-64bit
`uid:gid`: 502:20
<details><summary><code>pip_packages</code></summary>

<br>

absl-py==1.2.0
aiohttp==3.8.3
aiosignal==1.2.0
anyio==3.6.1
-e <git+ssh://git@bitbucket.org/bengrouptech/bento-tensorflow-poc.git@b8b217725656f5a2f7b9b4880f3dc1bfaa5296f8#egg=api&subdirectory=api>
appdirs==1.4.4
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asgiref==3.5.2
asttokens==2.0.8
astunparse==1.6.3
async-timeout==4.0.2
attrs==22.1.0
aws-cdk-lib==2.45.0
aws-cdk.aws-apigatewayv2-alpha==2.45.0a0
aws-cdk.aws-apigatewayv2-authorizers-alpha==2.45.0a0
aws-cdk.aws-apigatewayv2-integrations-alpha==2.45.0a0
aws-cdk.aws-lambda-python-alpha==2.45.0a0
Babel==2.10.3
backcall==0.2.0
beautifulsoup4==4.11.1
bentoml==1.0.7
black==22.10.0
bleach==5.0.1
boto3==1.24.89
botocore==1.27.89
Brotli==1.0.9
build==0.8.0
cachetools==5.2.0
cattrs==22.1.0
certifi==2022.9.24
cffi==1.15.1
charset-normalizer==2.1.1
circus==0.17.1
click==8.1.3
cloudpickle==2.2.0
commonmark==0.9.1
ConfigArgParse==1.5.3
constructs==10.1.130
contextlib2==21.6.0
contourpy==1.0.5
coverage==6.5.0
cycler==0.11.0
debugpy==1.6.3
decorator==5.1.1
deepmerge==1.0.1
defusedxml==0.7.1
Deprecated==1.2.13
entrypoints==0.4
exceptiongroup==1.0.0rc9
execnet==1.9.0
executing==1.1.1
fastjsonschema==2.16.2
Flask==2.2.2
Flask-BasicAuth==0.2.0
Flask-Cors==3.0.10
flatbuffers==22.9.24
fonttools==4.38.0
frozenlist==1.3.1
fs==2.4.16
gast==0.4.0
gevent==22.8.0
geventhttpclient==2.0.2
google-auth==2.12.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
greenlet==1.1.3.post0
grpcio==1.49.1
h11==0.14.0
h5py==3.7.0
-e <git+ssh://git@bitbucket.org/bengrouptech/bento-tensorflow-poc.git@b8b217725656f5a2f7b9b4880f3dc1bfaa5296f8#egg=iac&subdirectory=iac>
idna==3.4
imageio==2.22.2
iniconfig==1.1.1
ipykernel==6.16.0
ipython==8.5.0
ipython-genutils==0.2.0
itsdangerous==2.1.2
jedi==0.18.1
Jinja2==3.1.2
jmespath==1.0.1
jsii==1.69.0
json5==0.9.10
jsonschema==4.16.0
jupyter-core==4.11.1
jupyter-server==1.21.0
jupyter_client==7.4.2
jupyterlab==3.4.8
jupyterlab-pygments==0.2.2
jupyterlab_server==2.16.0
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.4
libclang==14.0.6
locust==2.12.1
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib==3.6.1
matplotlib-inline==0.1.6
mistune==2.0.4
-e <git+ssh://git@bitbucket.org/bengrouptech/bento-tensorflow-poc.git@b8b217725656f5a2f7b9b4880f3dc1bfaa5296f8#egg=model&subdirectory=model>
msgpack==1.0.4
multidict==6.0.2
mypy-extensions==0.4.3
nbclassic==0.4.5
nbclient==0.7.0
nbconvert==7.2.1
nbformat==5.7.0
nest-asyncio==1.5.6
notebook==6.5.1
notebook_shim==0.2.0
numpy==1.23.4
oauthlib==3.2.1
opencv-python==4.6.0.66
opentelemetry-api==1.12.0
opentelemetry-instrumentation==0.33b0
opentelemetry-instrumentation-aiohttp-client==0.33b0
opentelemetry-instrumentation-asgi==0.33b0
opentelemetry-sdk==1.12.0
opentelemetry-semantic-conventions==0.33b0
opentelemetry-util-http==0.33b0
opt-einsum==3.3.0
packaging==21.3
pandas==1.5.1
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.10.1
pep517==0.13.0
pexpect==4.8.0
phitoduck-projen==0.1.1
pickleshare==0.7.5
Pillow==9.2.0
pip-tools==6.9.0
platformdirs==2.5.2
pluggy==1.0.0
projen==0.63.17
prometheus-client==0.14.1
prompt-toolkit==3.0.31
protobuf==3.19.6
psutil==5.9.2
ptyprocess==0.7.0
publication==0.0.3
pure-eval==0.2.2
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pydantic==1.10.2
Pygments==2.13.0
pynvml==11.4.1
pyparsing==3.0.9
pyrsistent==0.18.1
pytest==7.1.3
pytest-asyncio==0.19.0
pytest-cov==4.0.0
pytest-forked==1.4.0
pytest-xdist==2.5.0
python-dateutil==2.8.2
python-dotenv==0.21.0
python-multipart==0.0.5
pytz==2022.5
PyYAML==6.0
pyzmq==24.0.1
requests==2.28.1
requests-oauthlib==1.3.1
rich==12.6.0
roundrobin==0.0.4
rsa==4.9
s3transfer==0.6.0
schema==0.7.5
scipy==1.9.3
Send2Trash==1.8.0
simple-di==0.1.5
six==1.16.0
sniffio==1.3.0
soupsieve==2.3.2.post1
stack-data==0.5.1
starlette==0.21.0
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.0
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.27.0
termcolor==2.0.1
terminado==0.16.0
tf-keras-vis==0.8.2
tinycss2==1.2.1
tomli==2.0.1
tornado==6.2
traitlets==5.5.0
typeguard==2.13.3
typing_extensions==4.4.0
urllib3==1.26.12
uvicorn==0.18.3
watchfiles==0.17.0
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.4.1
Werkzeug==2.2.2
wrapt==1.14.1
yarl==1.8.1
zope.event==4.5.0
zope.interface==5.5.0

</details>
I am ashamed of how long this req's list is 🤣. It's all the reqs for the service, the model, and the iac, which seems to add up to a lot.
a
the long import time might have to do with how long it takes to import cv2? everything seems fine to me. You can maybe try to move 
import cv2
lazily under the function
e
I added a print statement right after 
import cv2
and it seems to run instantly when I run 
bentoml serve --reload
. Here is a soup of logs, annotated describing the behavior:
Copy code
Imported CV2!

2022-10-27 01:14:58.948315: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-27T01:15:03-0600 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "." can be accessed at <http://localhost:3000/metrics>.

2022-10-27T01:15:04-0600 [INFO] [cli] Starting development HTTP BentoServer from "." running on <http://0.0.0.0:3000> (Press CTRL+C to quit)

2022-10-27 01:15:05 circus[92733] [INFO] Loading the plugin...
2022-10-27 01:15:05 circus[92733] [INFO] Endpoint: '<tcp://127.0.0.1:60975>'
2022-10-27 01:15:05 circus[92733] [INFO] Pub/sub: '<tcp://127.0.0.1:60976>'

2022-10-27T01:15:05-0600 [INFO] [observer] Watching directories: ['/Users/eric/repos/ben/bentoml-poc/api', '/Users/eric/bentoml/models']

Imported CV2!

2022-10-27 01:15:05.178569: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-27 01:15:09.652504: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

... proceeds to hang for 60-180 seconds, then the web UI becomes available and requests can go ahead
You can see there are tensorflow/keras logs interspersed with the BentoML logs. It seems like the call to 
bentoml.keras.get(...)
initializes some sort of keras process that then takes a lot of time to start up. I observe this same behavior when I run 
bentoml.keras.get()
from a jupyter notebook for the same model tag. The cell takes about 60-180 seconds to finish and then subsequent cells have access to the model.
How big is your model?
It seems to be 71 MB:
Copy code
$ du -sh ~/bentoml/models/tensorflow_ctr_keras                                                                         
 71M    tensorflow_ctr_keras
I have an x86 mac. I confirmed with a co-worker on an M1 mac that they experience the same thing.
a
Got it. Thanks for the information. I can take a look. Also cc @larme (shenyang)
s
I think it might actually be pretty valuable to have a mode where we start a runner server independently of the service and do reloads on the service as required, but unfortunately we don't support anything like that at the moment because it would be a decent amount of work to implement.
💡 1
Open in Slack
PreviousNext
Powered by