Nihar Rao
04/28/2025, 8:33 PMkubectl describe pod <JM pod> shows the pod restarted due to OOM as well.
3. After JM was OOMkilled, JM was restarted and 24 new taskmanagers pods were started and is confirmed on flink UI on available task slots section.
4. There was no impact on job (it restarted successfully) but there are 48 taskmanagers running out of which 24 are standby. The expected behaviour after a JM OOM with HA enabled is no starting of new task managers.
5. I simulated the OOM on JM pod for one of our other job but no similar behaviour was observed.
I have added the flink UI showing 24 extra TMs (48 task slots) screenshot and kubectl output in the thread.
Can you please help us on how to debug this as kubernetes operator don't show any relevant information on why this happened. Thanks!Nihar Rao
04/28/2025, 8:34 PMioi-quality-667f575877-btfkv 1/1 Running 1 (39h ago) 4d16h
ioi-quality-taskmanager-1-1 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-10 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-11 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-12 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-13 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-14 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-15 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-16 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-17 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-18 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-19 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-2 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-20 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-21 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-22 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-23 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-24 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-3 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-4 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-5 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-6 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-7 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-8 1/1 Running 0 4d16h
ioi-quality-taskmanager-1-9 1/1 Running 0 4d16h
ioi-quality-taskmanager-2-1 1/1 Running 0 39h
ioi-quality-taskmanager-2-10 1/1 Running 0 39h
ioi-quality-taskmanager-2-11 1/1 Running 0 39h
ioi-quality-taskmanager-2-12 1/1 Running 0 39h
ioi-quality-taskmanager-2-13 1/1 Running 0 39h
ioi-quality-taskmanager-2-14 1/1 Running 0 39h
ioi-quality-taskmanager-2-15 1/1 Running 0 39h
ioi-quality-taskmanager-2-16 1/1 Running 0 39h
ioi-quality-taskmanager-2-17 1/1 Running 0 39h
ioi-quality-taskmanager-2-18 1/1 Running 0 39h
ioi-quality-taskmanager-2-19 1/1 Running 0 39h
ioi-quality-taskmanager-2-2 1/1 Running 0 39h
ioi-quality-taskmanager-2-20 1/1 Running 0 39h
ioi-quality-taskmanager-2-21 1/1 Running 0 39h
ioi-quality-taskmanager-2-22 1/1 Running 0 39h
ioi-quality-taskmanager-2-23 1/1 Running 0 39h
ioi-quality-taskmanager-2-24 1/1 Running 0 39h
ioi-quality-taskmanager-2-3 1/1 Running 0 39h
ioi-quality-taskmanager-2-4 1/1 Running 0 39h
ioi-quality-taskmanager-2-5 1/1 Running 0 39h
ioi-quality-taskmanager-2-6 1/1 Running 0 39h
ioi-quality-taskmanager-2-7 1/1 Running 0 39h
ioi-quality-taskmanager-2-8 1/1 Running 0 39h
ioi-quality-taskmanager-2-9 1/1 Running 0 39hNihar Rao
04/28/2025, 8:34 PMNishant More
05/13/2025, 10:13 PMNihar Rao
05/13/2025, 10:28 PMNishant More
05/13/2025, 10:28 PMNishant More
05/13/2025, 10:38 PMNishant More
06/02/2025, 6:48 PMNihar Rao
06/02/2025, 6:54 PMNishant More
06/03/2025, 7:59 PM