Rushikesh Gulve
06/12/2025, 9:44 AMapiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: data-processing-flink
namespace: flink
spec:
image: 624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest
flinkVersion: "v1_20"
serviceAccount: flink
flinkConfiguration:
rest.port: "8081"
jobmanager.rpc.address: "data-processing-flink"
jobmanager.rpc.port: "6123"
taskmanager.numberOfTaskSlots: "4"
podTemplate:
spec:
imagePullSecrets:
- name: ecrscr-credentials
containers:
- name: flink-main-container
imagePullPolicy: Always
envFrom:
- secretRef:
name: redis-secret-main
env:
- name: KAFKA_BROKER
value: "kafka.flink.svc.cluster.local:9092"
- name: BOOTSTRAP_SERVERS
value: "kafka.flink.svc.cluster.local:9092"
- name: MINIO_SERVER
value: "http://myminio-hl.flink.svc.cluster.local:9000"
- name: MINIO_USER
value: "minio"
- name: MINIO_PASSWORD
value: "minio123"
- name: MAX_PARALLELISM
value: "128"
securityContext:
runAsUser: 9999
runAsGroup: 9999
jobManager:
resource:
memory: "2048m"
cpu: 0.4
taskManager:
resource:
memory: "4096m"
cpu: 2
replicas: 8
job:
entryClass: "org.apache.flink.client.python.PythonDriver"
args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
upgradeMode: stateless
Though I have set 4gb memory explicitly per Task Manager but it ends up using lot more than that and ultimately goes Out Of Memory. I have 2 questions:
Caused by: java.util.concurrent.TimeoutException: Invocation of [RemoteRpcInvocation(TaskExecutorGateway.submitTask(TaskDeploymentDescriptor, JobMasterId, Time))] at recipient [pekko.tcp://flink@192.168.164.1:6122/user/rpc/taskmanager_0] timed out. This is usually caused by: 1) Pekko failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase pekko.ask.timeout.
1. Can I set any limits to to resource utilization?
2. Is it normal to use this much memory or is there something I can do to reduce consumption?Antonio Davide Cali
06/12/2025, 9:46 AMRushikesh Gulve
06/12/2025, 9:51 AMPedro Mázala
06/12/2025, 9:55 AMRushikesh Gulve
06/12/2025, 10:01 AMPedro Mázala
06/12/2025, 10:03 AMRushikesh Gulve
06/12/2025, 10:13 AMPedro Mázala
06/12/2025, 1:39 PMtaskmanager.numberOfTaskSlots: "4" states you will have 4 tasks per TM. and if 1 task (parallelism = 1) drinks 4gb, you're setting Flink to use 4*4 at least (not considering the TM memory).
> Is python the issue here?
Not python per se. It could be the way you're leveraging your data within your python code. It could be something related to your window sizes, how much data you're consuming from your source in a single shot...
> because as far as my understanding goes State is evoked per key in the open function of any KeyedProcessFunction class. So this should start occupying space only once data enters the pipeline.
yeah, it will use the data as soon as you have data flowing through it. So, you're saying your source data is empty and still Flink is consuming 4gb? there is something fishy around that for sure.
> Is there any tool which I can use to get a clearer picture of resource utilization?
You could play with Java profilers to see where memory consumption is going. Take a look on Flink memory setup and there is also this blog post from Yaroslav about flink and profilersRushikesh Gulve
06/13/2025, 5:38 AM