Rushikesh Gulve
07/14/2025, 7:15 AMapiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
name: data-processing-flink
namespace: flink
spec:
image: <http://624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest|624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest>
flinkVersion: "v1_20"
serviceAccount: flink
flinkConfiguration:
rest.port: "8081"
jobmanager.rpc.address: "data-processing-flink"
jobmanager.rpc.port: "6123"
#taskmanager.numberOfTaskSlots: "2"
# High Availability
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.storageDir: file:///flink-data/ha
# Checkpoints and Savepoints
state.checkpoints.dir: file:///flink-data/checkpoints
state.savepoints.dir: file:///flink-data/savepoints
state.backend.type: rocksdb
state.backend.rocksdb.memory.managed: "true" # Enable Flink's memory management for RocksDB
state.backend.rocksdb.memory.write-buffer-size: "64mb" # Limit write buffer size
state.backend.rocksdb.memory.max-file-size-to-compact: "2mb" # Increase compaction file size
state.backend.rocksdb.memory.min-files-to-compact: "3" # Reduce compaction frequency
state.backend.rocksdb.memory.max-files-to-compact: "10"
state.backend.rocksdb.memory.max-output-file-size: "128mb"
state.backend.rocksdb.memory.cache-capacity: "256mb" # Reduce cache size
state.backend.rocksdb.memory.managed: true
state.backend.rocksdb.memory.write-buffer-ratio: "0.5"
state.backend.rocksdb.memory.high-prio-pool-ratio: "0.1"
python.fn-execution.bundle.size: "100"
python.fn-execution.memory.managed: "true"
python.fn-execution.buffer.memory.size: "64mb"
#execution.checkpointing.alignment-timeout: "0" # ms, after which checkpoint switches to unaligned mode dynamically
#execution.checkpointing.unaligned: "true" # Enable unaligned checkpoints
#rest.profiling.enabled: "true"
#env.java.opts.taskmanager: "-Dorg.apache.beam.sdk.fn=DEBUG -Dorg.apache.beam.runners.fnexecution.control.FnApiControlClient=DEBUG"
podTemplate:
spec:
imagePullSecrets:
- name: ecrscr-credentials
containers:
- name: flink-main-container
imagePullPolicy: Always
envFrom:
- secretRef:
name: redis-secret-main
env:
- name: _PYTHON_WORKER_MEMORY_LIMIT
value: "350000000"
- name: KAFKA_BROKER
value: "kafka.flink.svc.cluster.local:9092"
- name: BOOTSTRAP_SERVERS
value: "kafka.flink.svc.cluster.local:9092"
- name: MINIO_SERVER
value: "<http://myminio-hl.flink.svc.cluster.local:9000>"
- name: MINIO_USER
value: "minio"
- name: MINIO_PASSWORD
value: "minio123"
- name: MAX_PARALLELISM
value: "128"
volumeMounts:
- mountPath: /flink-data
name: flink-pvc
volumes:
- name: flink-pvc
persistentVolumeClaim:
claimName: flink-data-pvc
securityContext:
runAsUser: 9999
runAsGroup: 9999
jobManager:
resource:
memory: "3072m"
cpu: 3
taskManager:
resource:
memory: "4096m"
cpu: 1
replicas: 32
job:
entryClass: "org.apache.flink.client.python.PythonDriver"
args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
upgradeMode: stateless
I have disabled checkpointing purposely.
The issue is I have kept parallelism as 32 and have multiple processes. Now I am seeing in all the task manager logs that the process is completed simultaneously but in kafka partitions the data is received one after another with lot of delay. My output is only a single data point sinked to kafka which is link to a s3 bucket. Why am I not seeing all the data in kafka partitions simultaneously is data is processed simultaneously. (the parallelism is utilized by producing different keys to different partitions)
KafkaSink.builder()
.set_bootstrap_servers(BOOTSTRAP_SERVERS)
.set_record_serializer(
KafkaRecordSerializationSchema.builder()
.set_topic(topic)
.set_value_serialization_schema(
JsonRowSerializationSchema.builder()
.with_type_info(type_info)
.build()
)
.build()
)
.set_delivery_guarantee(DeliveryGuarantee.AT_LEAST_ONCE)
.build()
this is my kafka sink.
I am using flink 1.20
Any help is greatly appreciated. Thankyou.Thomas Cooper
07/15/2025, 8:27 AMRushikesh Gulve
07/17/2025, 12:07 PM