Can I check what new directories does bentoml 1.3 ...
# ask-for-help
j
Can I check what new directories does bentoml 1.3 need to write to, as compared to bentoml 1.2? I tried upgrading to bentoml 1.3 and my bento with the exact same code will not start, it seems to keep restarting the bento server with no errors in the log. In my production environment, we need to run the container as a read only file system on kubernetes. When debugging, I downloaded the bento to my local machine and used the following docker compose:
Copy code
services:
  bento:
    build:
      context: iris-clf
      dockerfile: env/docker/Dockerfile
    ports:
      - "3000:3000"
    read_only: true
    volumes:
      - ./tempdir/bentos:/home/bentoml/bentos
      - ./tempdir/models:/home/bentoml/models
      - ./tempdir/envs:/home/bentoml/envs
      - ./tempdir/tmp:/home/bentoml/tmp
      - ./tempdir/.cache:/home/bentoml/.cache
      - ./tempdir/tmp:/tmp
The above volume mounts are the ones that are required in bentoml 1.2. However, for bentos built with bentoml 1.3, the above volume mount is not sufficient. The bento starts normally if I set
read_only
to false.
j
CC: @Frost Ming
f
/home/bentoml/task_result.db
is a file
why you create volumes for sub directories and files
j
because the model is saved inside
/home/bentoml/models
and
/home/bentoml/src
, so if i mount emptydir into
/home/bentoml
, my models that are saved inside the bento will not be there. Not sure if i can specify a different directory to save the models to.
if i mount a volume and set the
BENTOML_RESULT_STORE
there, the problem is slightly different - the bento server will not repeatedly restart, but
<http://localhost:3000>
returns
Internal Server Error
, but the bento's logs do not show any error.
Copy code
services:
  bento:
    build:
      context: iris-clf
      dockerfile: env/docker/Dockerfile
    ports:
      - "3000:3000"
    read_only: true
    environment:
      - BENTOML_RESULT_STORE=/home/bentoml/result-store/task_result.db
    volumes:
      - ./tempdir/bentos:/home/bentoml/bentos
      - ./tempdir/models:/home/bentoml/models
      - ./tempdir/envs:/home/bentoml/envs
      - ./tempdir/tmp:/home/bentoml/tmp
      - ./tempdir/.cache:/home/bentoml/.cache
      - ./tempdir/tmp:/tmp
      - ./tempdir/result-store:/home/bentoml/result-store
f
The result store would be deleted when server starts, try putting it under a directory and mount that directory
And that file needs to be writable
j
ok this docker compose seems to work, where the result store directory is mounted as
rw
Copy code
services:
  bento:
    build:
      context: iris-clf
      dockerfile: env/docker/Dockerfile
    ports:
      - "3000:3000"
    read_only: true
    environment:
      - BENTOML_RESULT_STORE=/home/bentoml/bentoml-result-store/task_result.db
    volumes:
      - ./tempdir/bentos:/home/bentoml/bentos
      - ./tempdir/models:/home/bentoml/models
      - ./tempdir/envs:/home/bentoml/envs
      - ./tempdir/tmp:/home/bentoml/tmp
      - ./tempdir/.cache:/home/bentoml/.cache
      - ./tempdir/prometheus_multiproc_dir:/home/bentoml/prometheus_multiproc_dir
      - ./tempdir/tmp:/tmp
      - ./tempdir/result-store:/home/bentoml/bentoml-result-store:rw
can confirm that mounting an emptyDir somewhere and then setting the env var
BENTOML_RESULT_STORE
to be inside that directory works in my production environment (k8s cluster) Thanks for the help!