Hi everyone, I am trying to deploy Flink using Kub...
# troubleshooting
r
Hi everyone, I am trying to deploy Flink using Kubernetes operator.
Copy code
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: data-processing-flink
  namespace: flink
spec:
  image: 624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest
  flinkVersion: "v1_20"
  serviceAccount: flink
  flinkConfiguration:
    rest.port: "8081"
    jobmanager.rpc.address: "data-processing-flink"
    jobmanager.rpc.port: "6123"
    taskmanager.numberOfTaskSlots: "4"
  podTemplate:
    spec:
      imagePullSecrets:
        - name: ecrscr-credentials
      containers:
        - name: flink-main-container
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: redis-secret-main
          env:
            - name: KAFKA_BROKER
              value: "kafka.flink.svc.cluster.local:9092"
            - name: BOOTSTRAP_SERVERS
              value: "kafka.flink.svc.cluster.local:9092"
            - name: MINIO_SERVER
              value: "http://myminio-hl.flink.svc.cluster.local:9000"
            - name: MINIO_USER
              value: "minio"
            - name: MINIO_PASSWORD
              value: "minio123"
            - name: MAX_PARALLELISM
              value: "128"
      securityContext:
        runAsUser: 9999
        runAsGroup: 9999
  jobManager:
    resource:
      memory: "2048m"
      cpu: 0.4
  taskManager:
    resource:
      memory: "4096m"
      cpu: 2
    replicas: 8
  job:
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
    upgradeMode: stateless
Though I have set 4gb memory explicitly per Task Manager but it ends up using lot more than that and ultimately goes Out Of Memory. I have 2 questions:
Copy code
Caused by: java.util.concurrent.TimeoutException: Invocation of [RemoteRpcInvocation(TaskExecutorGateway.submitTask(TaskDeploymentDescriptor, JobMasterId, Time))] at recipient [pekko.tcp://flink@192.168.164.1:6122/user/rpc/taskmanager_0] timed out. This is usually caused by: 1) Pekko failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase pekko.ask.timeout.
1. Can I set any limits to to resource utilization? 2. Is it normal to use this much memory or is there something I can do to reduce consumption?
a
Hello 🙂 what you set-up is not the limit of the resource of TaskManager, but how much resource you are providing to each TaskManager Pod, so if job use much more memory and you didn't set a checkpoint strategy, it's very likely a OOM would happen
☝️ 1
r
@Antonio Davide Cali @Pedro Mázala Thanks for the reply. is there any option to set the limits, am not able to find that in the official documentation. how does setting up checkpoint strategy, help with resource utilisation
p
What you'll have to do is understand where your application is blowing. Perhaps a too big of a window, state or things like this.
r
@Pedro Mázala Thanks for your response. I had one small doubt. The error that I am getting are at the very start of deployment before any data is produced at the source. The initial setup itself is blowing up. How can my application consume memory for states and windows before any data is produced?
p
Perhaps something on the python load up? It's hard to tell without knowing the code. I had pipelines running sometimes with 1gb of memory processing a decent amount of data, so I think flink side should be safe. When you run it locally, what is the memory behavior?
r
@Pedro Mázala On running locally for parallelism=1, I could see memory utilization spiking by around 3.5-4gb. I was producing data with 1 key for this. On kubernetes I scheduled 8 TM pods with 4 TS each. My plan was to produce similar data with 32 keys to leverage parallelism. Is python the issue here? because as far as my understanding goes State is evoked per key in the open function of any KeyedProcessFunction class. So this should start occupying space only once data enters the pipeline. Is there any tool which I can use to get a clearer picture of resource utilization?
p
> On running locally for parallelism=1, I could see memory utilization spiking by around 3.5-4gb If this is the case, your TM will certainly use more memory than that. The configuration
taskmanager.numberOfTaskSlots: "4"
states you will have 4 tasks per TM. and if 1 task (parallelism = 1) drinks 4gb, you're setting Flink to use 4*4 at least (not considering the TM memory). > Is python the issue here? Not python per se. It could be the way you're leveraging your data within your python code. It could be something related to your window sizes, how much data you're consuming from your source in a single shot... > because as far as my understanding goes State is evoked per key in the open function of any KeyedProcessFunction class. So this should start occupying space only once data enters the pipeline. yeah, it will use the data as soon as you have data flowing through it. So, you're saying your source data is empty and still Flink is consuming 4gb? there is something fishy around that for sure. > Is there any tool which I can use to get a clearer picture of resource utilization? You could play with Java profilers to see where memory consumption is going. Take a look on Flink memory setup and there is also this blog post from Yaroslav about flink and profilers
r
@Pedro Mázala Thankyou for your response.