hi all, i am using rocks db state backend for storing around 73 gb of total state. use case is to keep updating the state in rocks db from two sources unless there is a match in another source based on key when they are emited out to sink i have setup a custom source which broadcast an event which is then read by broadcast function and emits the entire state out also clears the state. the storing and processing is done in keyed broadcast function source 1 (kafka stream producing 40 million at a time) parallelism is 3 if i increase it causes a lot of back pressure source 2(kafka producing 2 million event at a time ) parallelism is 1 broadcast function operator parallelism is 40 sink (kafka) parallelism is 35 the process runs fine until 20 million records and then for a duration of 3-4 minutes till 28 million is reached there is high back pressure in source one because of cpu utilization of 100% in broadcast function. below is my config with 20 task managers job manager is at 1.5 cpu and 1 gb ram and the task manager is at 4 cpu and 10 gb ram. even if i disable check pointing i see the same trend what i am assuming is i have lot of lookups read and writes to rocks db as per my use case that might be causing this issue .. but what i have failed to identify is why is it for a small interval of time and what i have seen is sometimes it crashes the task manager causing it to restart the system. i tried setting up block-cache config but that also dint help.. any suggesstion to overcome this?

"taskmanager.numberOfTaskSlots": "2"
    "state.backend.rocksdb.timer-service.factory": "HEAP"
    "state.backend": "rocksdb"
    "state.backend.incremental": "true"
    "state.backend.rocksdb.localdir": "/tmp/data1,/tmp/data2,/tmp/data3,/tmp/data4,/tmp/data5,/tmp/data4,/tmp/data6,/tmp/data7,/tmp/data8,/tmp/data9,/tmp/data10"
    "state.backend.rocksdb.ttl.compaction.filter.enabled": "true"
    "state.backend.rocksdb.memory.managed": "true"
    "cluster.evenly-spread-out-slots": "true"
    "taskmanager.memory.managed.size": "7g"
    "taskmanager.network.memory.buffer-debloat.enabled": "true"
    "taskmanager.network.max-num-tcp-connections": "2"

What kind of disk do you use for RocksDB state?

thanks for reply @sap1ens we are using gp2

Try using instance-level storage, not EBS. It can make huge difference.

i also tried in memory rocks db with saving state for my daily load around 30 million across task manager in memory..but that too showed similar results.