https://flink.apache.org/ logo
Join Slack
Powered by
Hi team, we are using `slotmanager.redundant-taskm...
# troubleshooting
j
Hi team, we are using 
slotmanager.redundant-taskmanager-num: "1"
to run a redundant TM to speed up recovery. But what we have seen is flink is keep trying to schedule a new TM pod until redundant TM is ready every few seconds. So for some time we are seeing many extra TMs in Pending state and it is triggering unnecessary node autoscaling. Wondering if there is a configured interval for flink to detect if redundant TM is in Ready state. We are using flink operator for deployment.
During upgrade
Copy code
xxxx-flink-6f5bd58466-5bzcn   1/1     Running             0                7m40s
xxxx-flink-6f5bd58466-tbqnq   1/1     Running             0                7m40s
xxxx-flink-taskmanager-1-1    1/1     Running             0                6m38s
xxxx-flink-taskmanager-1-10   0/1     Pending             0                3m12s
xxxx-flink-taskmanager-1-11   0/1     Pending             0                2m42s
xxxx-flink-taskmanager-1-12   0/1     Pending             0                2m12s
xxxx-flink-taskmanager-1-13   0/1     Pending             0                102s
xxxx-flink-taskmanager-1-14   0/1     Pending             0                72s
xxxx-flink-taskmanager-1-15   0/1     ContainerCreating   0                42s
xxxx-flink-taskmanager-1-16   0/1     ContainerCreating   0                12s
xxxx-flink-taskmanager-1-2    1/1     Running             0                6m38s
xxxx-flink-taskmanager-1-3    0/1     Pending             0                6m38s
xxxx-flink-taskmanager-1-4    1/1     Running             0                6m13s
xxxx-flink-taskmanager-1-5    0/1     Pending             0                5m42s
xxxx-flink-taskmanager-1-6    0/1     Pending             0                5m12s
xxxx-flink-taskmanager-1-7    0/1     Pending             0                4m42s
xxxx-flink-taskmanager-1-8    0/1     Pending             0                4m12s
xxxx-flink-taskmanager-1-9    0/1     Pending             0                3m42s
Once deployment is stabilized
Copy code
xxxx-flink-6f5bd58466-5bzcn   1/1     Running     0                12m
xxxx-flink-6f5bd58466-tbqnq   1/1     Running     0                12m
xxxx-flink-taskmanager-1-1    1/1     Running     0                11m
xxxx-flink-taskmanager-1-18   1/1     Running     0                4m3s
xxxx-flink-taskmanager-1-2    1/1     Running     0                11m
xxxx-flink-taskmanager-1-4    1/1     Running     0                11m
Open in Slack
PreviousNext
Powered by