Hi everyone, I am trying to deploy my pyflink appl...
# troubleshooting
r
Hi everyone, I am trying to deploy my pyflink application using Kubernetes operator. I have the system running stable locally, however, when trying to deploy it on Kubernetes, I am facing issue with configuration of RocksDB state backend. I observed, the time taken to fetch any state is too large and is becoming the bottleneck. While debugging, I have configured a prometheus exporter to collect the metrics. This is the graph that I am getting for certain metrics of RocksDB.
Copy code
state.backend.rocksdb.memory.managed: "true"
    state.backend.rocksdb.memory.write-buffer-ratio: "0.4"  # 40% for write buffers
    state.backend.rocksdb.memory.high-prio-pool-ratio: "0.3"  # 30% for frequently accessed data
    
    # Large block cache to keep all data in memory
    state.backend.rocksdb.block.cache-size: "256m"  # Enough for 6x your state size
    state.backend.rocksdb.write-buffer.size: "128m"  # Large write buffers
    
    # Optimize for read performance (your use case)
    state.backend.rocksdb.bloom-filter.bits-per-key: "10"
    state.backend.rocksdb.bloom-filter.block-based-mode: "true"
    
    # Minimize disk writes - keep everything in memory as long as possible
    state.backend.rocksdb.compaction.level.target-file-size-base: "64mb"
    state.backend.rocksdb.compaction.level.max-size-level-base: "256mb"
    
    # Fast compression for checkpoints only
    state.backend.rocksdb.compression.type: "NO_COMPRESSION"
    
    # Local storage for RocksDB working files
    state.backend.rocksdb.localdir: /tmp/rocksdb
This is my backend related configurations that I have done. (I was expecting to keep a high percentage in memory for quick fetch) I am not able to understand the root cause using metrics. Can anyone guide regarding how to troubleshoot this issue.