Rushikesh Gulve
07/22/2025, 11:42 AMmanaged_memory , block_cache but none of them is helping. After experimenting I observed that serialization/deserialization of state when using RockDB is the bottleneck slowing down the process and we cannot bypass it when using pyflink.
My question is how can I use state in flink if I want a fault tolerant state storage which high throughput.
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
name: data-processing-flink
namespace: flink
spec:
image: <http://624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest|624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest>
flinkVersion: "v1_20"
serviceAccount: flink
flinkConfiguration:
rest.port: "8081"
jobmanager.rpc.address: "data-processing-flink"
jobmanager.rpc.port: "6123"
env.java.opts.taskmanager: >
-Dorg.apache.beam.sdk.fn=DEBUG
-Dorg.apache.beam.runners.fnexecution.control.FnApiControlClient=DEBUG
# High Availability
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.storageDir: file:///flink-data/ha
# Checkpoints and Savepoints - FAULT TOLERANT
state.checkpoints.dir: file:///flink-data/checkpoints
state.savepoints.dir: file:///flink-data/savepoints
#execution.checkpointing.interval: "300000" # 5 minutes
#execution.checkpointing.mode: EXACTLY_ONCE
# RocksDB Backend - MEMORY OPTIMIZED FOR SPEED
state.backend.type: rocksdb
#state.backend.incremental: "true"
state.backend.rocksdb.memory.managed: "true"
state.backend.rocksdb.memory.write-buffer-ratio: "0.4"
state.backend.rocksdb.memory.high-prio-pool-ratio: "0.3"
state.backend.rocksdb.block.cache-size: "512m"
state.backend.rocksdb.write-buffer.size: "128m"
# Optimize for read performance (your use case)
state.backend.rocksdb.bloom-filter.bits-per-key: "10"
state.backend.rocksdb.bloom-filter.block-based-mode: "true"
state.backend.rocksdb.compaction.level.target-file-size-base: "64mb"
state.backend.rocksdb.compaction.level.max-size-level-base: "256mb"
state.backend.rocksdb.compression.type: "LZ4_COMPRESSION"
state.backend.rocksdb.localdir: /tmp/rocksdb
taskmanager.memory.managed.fraction: 0.3
taskmanager.memory.network.fraction: 0.1
taskmanager.memory.jvm-overhead.fraction: 0.1
# Remaining ~50% = 2GB for JVM heap
python.fn-execution.bundle.size: "1000"
python.fn-execution.memory.managed: "true"
python.fn-execution.buffer.memory.size: "128mb"
podTemplate:
spec:
imagePullSecrets:
- name: ecrscr-credentials
containers:
- name: flink-main-container
imagePullPolicy: Always
envFrom:
- secretRef:
name: redis-secret-main
env:
- name: _PYTHON_WORKER_MEMORY_LIMIT
value: "350000000"
- name: KAFKA_BROKER
value: "kafka.flink.svc.cluster.local:9092"
- name: BOOTSTRAP_SERVERS
value: "kafka.flink.svc.cluster.local:9092"
- name: MINIO_SERVER
value: "<http://myminio-hl.flink.svc.cluster.local:9000>"
- name: MINIO_USER
value: "minio"
- name: MINIO_PASSWORD
value: "minio123"
- name: MAX_PARALLELISM
value: "128"
# JVM tuning for optimal memory usage
- name: FLINK_ENV_JAVA_OPTS_TASKMANAGER
value: >
-XX:+UseG1GC
-XX:MaxGCPauseMillis=50
-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap
volumeMounts:
- mountPath: /flink-data
name: flink-pvc
# Resource limits for memory safety
resources:
requests:
memory: "4Gi"
cpu: "1"
limits:
memory: "4.2Gi" # Small buffer to prevent OOM kills
cpu: "1.5"
volumes:
- name: flink-pvc
persistentVolumeClaim:
claimName: flink-data-pvc
securityContext:
runAsUser: 9999
runAsGroup: 9999
jobManager:
resource:
memory: "3072m"
cpu: 3
taskManager:
resource:
memory: "4096m"
cpu: 1
replicas: 7
job:
entryClass: "org.apache.flink.client.python.PythonDriver"
args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
upgradeMode: stateless
This is my current configuration file. I have configured to aggressively use caching. Is there any better way to use states efficient. Any help would be greatly appreciated.