What's the `_result_store` ( a `Sqlite3Store`) on ...
# ask-for-help
p
What's the
_result_store
( a
Sqlite3Store
) on a
ServiceAppFactory
? Or more generally, why is a BentoML storing data locally in a sqlite database ? Is this needed to run a service, can we disable it ?
c
p
Ok, we are getting sporadic
sqlite3.OperationalError: database is locked
on the
CREATE TABLE IF NOT EXISTS result ...
query. I'm guessing this is a race condition when the service is created in multiple processes. As they all init the service,
Sqlite3Store
instances are created and entered (
__aenter__
is called) at the same time. So two processes try to create the DB file and/or the table, and one of them is kicked out by the lock.
c
Could you share a bit more about your setup? Do you run multiple BentoML serve instances on the same host?
p
We built a container image with
bentoml containerize
and deploy the produced image on a kubernetes cluster.
We do run multiple containers n the same host, but they should each get their own filesystem. We do run two workers in each container, so as I understand it we do have two processes creating their own service instance.
c
cc @Frost Ming @Jiang
j
Got
f
okay, I will move the init_db to
on_deployment
to avoid race condition
j
Hello @Pierre Buyle, thank you for helping us to spot this. We have released a PR to fix this, https://github.com/bentoml/BentoML/pull/5369 once it is merged, you can install from the
main
branch to test it out. We will issue a new release very promptly. Meanwhile, unfortunately you will have to limit the workers to 1 for now.
👍 1
f
FYI bentoml v1.4.15 is released with the fixes