We have a simple test pipeline that looks like this. I've been observing the storage space taken by RocksDB

kubernetes.ephemeral_storage.usage

and when the pipeline has been running for over a day every TM has about 20 GB of storage used. I'm trying to understand if: • This is "normal" and the pipeline just has a large amount of state • If one of our operators is keeping state indefinitely or • If our RocksDB is not clearing out old state properly. I have been noticing that the

flatmap

operator's state has only been increase, only have 1 hour of data as we just enabled the rocksDB metric (

state.backend.rocksdb.metrics.estimate-num-keys: true

). I saw in this guide to turn on some other RocksDB metrics, so we are waiting for that as well. I was wondering if anyone else has encountered this situation before?