Hi everyone! We are running into a weird issue wi...
# troubleshooting
n
Hi everyone! We are running into a weird issue with apache flink kubernetes operator 1.10.0 and apache flink 1.19.1. We are running jobs using native kubernetes mode and FlinkDeployment CRD. We are running a job with 24 taskmanagers and 1 Jobmanager replica with HA enabled Below is the chronological summary of events: 1. Job was initially started with 24 task managers. 2. JM pod was OOMkilled and it is confirmed by our KSM metrics and
kubectl describe pod <JM pod>
shows the pod restarted due to OOM as well. 3. After JM was OOMkilled, JM was restarted and 24 new taskmanagers pods were started and is confirmed on flink UI on available task slots section. 4. There was no impact on job (it restarted successfully) but there are 48 taskmanagers running out of which 24 are standby. The expected behaviour after a JM OOM with HA enabled is no starting of new task managers. 5. I simulated the OOM on JM pod for one of our other job but no similar behaviour was observed. I have added the flink UI showing 24 extra TMs (48 task slots) screenshot and kubectl output in the thread. Can you please help us on how to debug this as kubernetes operator don't show any relevant information on why this happened. Thanks!
kubectl get pod ouput:
Copy code
ioi-quality-667f575877-btfkv 1/1 Running 1 (39h ago) 4d16h
ioi-quality-taskmanager-1-1 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-10 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-11 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-12 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-13 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-14 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-15 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-16 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-17 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-18 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-19 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-2 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-20 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-21 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-22 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-23 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-24 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-3 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-4 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-5 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-6 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-7 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-8 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-9 1/1 Running 0 4d16h
ioi-quality-taskmanager-2-1 1/1 Running 0 39h
ioi-quality-taskmanager-2-10 1/1 Running 0 39h
ioi-quality-taskmanager-2-11 1/1 Running 0 39h
ioi-quality-taskmanager-2-12 1/1 Running 0 39h
ioi-quality-taskmanager-2-13 1/1 Running 0 39h
ioi-quality-taskmanager-2-14 1/1 Running 0 39h
ioi-quality-taskmanager-2-15 1/1 Running 0 39h
ioi-quality-taskmanager-2-16 1/1 Running 0 39h
ioi-quality-taskmanager-2-17 1/1 Running 0 39h
ioi-quality-taskmanager-2-18 1/1 Running 0 39h
ioi-quality-taskmanager-2-19 1/1 Running 0 39h
ioi-quality-taskmanager-2-2 1/1 Running 0 39h
ioi-quality-taskmanager-2-20 1/1 Running 0 39h
ioi-quality-taskmanager-2-21 1/1 Running 0 39h
ioi-quality-taskmanager-2-22 1/1 Running 0 39h
ioi-quality-taskmanager-2-23 1/1 Running 0 39h
ioi-quality-taskmanager-2-24 1/1 Running 0 39h
ioi-quality-taskmanager-2-3 1/1 Running 0 39h
ioi-quality-taskmanager-2-4 1/1 Running 0 39h
ioi-quality-taskmanager-2-5 1/1 Running 0 39h
ioi-quality-taskmanager-2-6 1/1 Running 0 39h
ioi-quality-taskmanager-2-7 1/1 Running 0 39h
ioi-quality-taskmanager-2-8 1/1 Running 0 39h
ioi-quality-taskmanager-2-9 1/1 Running 0 39h
flink UI screenshot showing 24 extra TMs
n
Hi @Nihar Rao did youo get any insught on this. I am facing same issue while running flink-1.18.1 on yarn cluster.
n
No, I had to create a JiRA ticket
n
oh ok. Could you provide the link to me?
Hi @Nihar Rao Is it ok if I add my observation also in the ticket?
n
sure go ahead! @Nishant More
n
done. thanks