Hi Team, the size of the data in the `BlobStorage`...
# troubleshooting
r
Hi Team, the size of the data in the
BlobStorage
folder in our
tmp
directory is already up to 400G, what will happen to delete this data? We deployed directly with docker compose, version 1.15.2.
d
Disaster! but it depends on how you are handling save points within.
If the blob contains checkpoint or save point data you could lose state.
What do you have for externalize.checkpoint.retention setting?
In any event before deleting something like this you should back up the folder first. That’s for certain.
✅ 1
Review checkpoint settings/dir etc ie checkpointing.internal, checkpointing.mode, and state.checkpoints.dir. Also without knowing all these things no one here can say for sure the consequences.
r
What do you have for externalize.checkpoint.retention setting?
Hi @D. Draco O'Brien. Thank you very much for your reply. Our checkpoint-related variables are not set, and I see that the default value of the
externalize.checkpoint.retention
variable is NO_EXTERNALIZED_CHECKPOINTS. So it shouldn’t be a problem with this setup?
d
Are you checkpointing production data anywhere?
The setting you mentioned decreased the likelihood of issues, but there is still the question of what is filling up the directory with this data? Also In-Progress checkouts could still be effected.
So you might try to examine the data and to understand why its filling up.
r
Review checkpoint settings/dir etc ie checkpointing.internal, checkpointing.mode, and state.checkpoints.dir. Also without knowing all these things no one here can say for sure the consequences.
About
checkpointing.internal
,
state.checkpoints.dir.
of configurations are configured according to different jobs, currently most of the jobs are c`heckpointing.internal` for 3min,
state.checkpoints.dir
are defined in
/opt/fink-cp
d
I think if you have large amounts of data and do frequent checkpointing it might still be stored even if you do not have this setting which is essentially for long term storage.
What is your setting for state.backend? rocksdb? #that would be my guess based on the use of the tmp directory.
r
What is your setting for state.backend? rocksdb? #that would be my guess based on the use of the tmp directory.
We do not set the
state.backend
configuration. It should be default value.