Hi everyone, I am trying to deploy Pyflink applica...
# troubleshooting
r
Hi everyone, I am trying to deploy Pyflink application using Kubernetes operator. I want to use state to store and fetch data. I tried multiple configuration for doing it. First I tried using Hashmap, it worked perfectly fine but it is not scalable. Then I tried RockDB, but it is becoming a bottleneck as the speed of state access is becoming significantly low (in order of seconds). I tried multiple configuration using
managed_memory
,
block_cache
but none of them is helping. After experimenting I observed that serialization/deserialization of state when using RockDB is the bottleneck slowing down the process and we cannot bypass it when using pyflink. My question is how can I use state in flink if I want a fault tolerant state storage which high throughput.
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: data-processing-flink
  namespace: flink
spec:
  image: <http://624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest|624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest>
  flinkVersion: "v1_20"
  serviceAccount: flink
  flinkConfiguration:
    rest.port: "8081"
    jobmanager.rpc.address: "data-processing-flink"
    jobmanager.rpc.port: "6123"

    env.java.opts.taskmanager: >
      -Dorg.apache.beam.sdk.fn=DEBUG
      -Dorg.apache.beam.runners.fnexecution.control.FnApiControlClient=DEBUG
 
    # High Availability
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: file:///flink-data/ha

    # Checkpoints and Savepoints - FAULT TOLERANT
    state.checkpoints.dir: file:///flink-data/checkpoints
    state.savepoints.dir: file:///flink-data/savepoints
    #execution.checkpointing.interval: "300000"  # 5 minutes
    #execution.checkpointing.mode: EXACTLY_ONCE
    
    # RocksDB Backend - MEMORY OPTIMIZED FOR SPEED
    state.backend.type: rocksdb
    #state.backend.incremental: "true" 
    
    state.backend.rocksdb.memory.managed: "true"
    state.backend.rocksdb.memory.write-buffer-ratio: "0.4"      
    state.backend.rocksdb.memory.high-prio-pool-ratio: "0.3" 
    
    state.backend.rocksdb.block.cache-size: "512m"  
    state.backend.rocksdb.write-buffer.size: "128m"  
    
    # Optimize for read performance (your use case)
    state.backend.rocksdb.bloom-filter.bits-per-key: "10"
    state.backend.rocksdb.bloom-filter.block-based-mode: "true"
    
    state.backend.rocksdb.compaction.level.target-file-size-base: "64mb"
    state.backend.rocksdb.compaction.level.max-size-level-base: "256mb"
    
    state.backend.rocksdb.compression.type: "LZ4_COMPRESSION"
    
    state.backend.rocksdb.localdir: /tmp/rocksdb
    
    taskmanager.memory.managed.fraction: 0.3 
    taskmanager.memory.network.fraction: 0.1 
    taskmanager.memory.jvm-overhead.fraction: 0.1 
    # Remaining ~50% = 2GB for JVM heap
    
    python.fn-execution.bundle.size: "1000"  
    python.fn-execution.memory.managed: "true"
    python.fn-execution.buffer.memory.size: "128mb"

  podTemplate:
    spec:
      imagePullSecrets:
        - name: ecrscr-credentials
      containers:
        - name: flink-main-container
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: redis-secret-main
          env:
            - name: _PYTHON_WORKER_MEMORY_LIMIT
              value: "350000000"
            - name: KAFKA_BROKER
              value: "kafka.flink.svc.cluster.local:9092"
            - name: BOOTSTRAP_SERVERS
              value: "kafka.flink.svc.cluster.local:9092"
            - name: MINIO_SERVER
              value: "<http://myminio-hl.flink.svc.cluster.local:9000>"
            - name: MINIO_USER
              value: "minio"
            - name: MINIO_PASSWORD
              value: "minio123"
            - name: MAX_PARALLELISM
              value: "128"
            # JVM tuning for optimal memory usage
            - name: FLINK_ENV_JAVA_OPTS_TASKMANAGER
              value: >
                -XX:+UseG1GC 
                -XX:MaxGCPauseMillis=50 
                -XX:+UnlockExperimentalVMOptions 
                -XX:+UseCGroupMemoryLimitForHeap
          volumeMounts:
            - mountPath: /flink-data
              name: flink-pvc
          # Resource limits for memory safety
          resources:
            requests:
              memory: "4Gi"
              cpu: "1"
            limits:
              memory: "4.2Gi"  # Small buffer to prevent OOM kills
              cpu: "1.5"
      volumes:
        - name: flink-pvc
          persistentVolumeClaim:
            claimName: flink-data-pvc
      securityContext:
        runAsUser: 9999
        runAsGroup: 9999

  jobManager:
    resource:
      memory: "3072m"
      cpu: 3

  taskManager:
    resource:
      memory: "4096m"
      cpu: 1
    replicas: 7

  job:
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
    upgradeMode: stateless
This is my current configuration file. I have configured to aggressively use caching. Is there any better way to use states efficient. Any help would be greatly appreciated.