Hi team, is it possible to provision standby taskm...
# troubleshootingj
Jeesmon Jacob
03/09/2023, 3:58 PMHi team, is it possible to provision standby taskmanagers with FlinkDeployment for quick recovery?
馃憖 1
Jeesmon Jacob
03/09/2023, 3:58 PMRef: https://developer.here.com/documentation/pipeline/dev_guide/topics/highly-available-pipelines.html
Provision Extra Task Managers with a Short Restart Strategyg
Gyula F贸ra
03/09/2023, 4:54 PMWith standalone mode , probably , but not with native mode as far as I know :)
Gyula F贸ra
03/09/2023, 4:54 PMBut it would be a nice feature to add to Flink for the native mode :)
j
Jeesmon Jacob
03/09/2023, 4:56 PMI tried with standalone but it also resets parallelism based on TM replicas 馃槥
g
Gyula F贸ra
03/09/2023, 4:57 PMAh yea right, those are connected with a single equation
j
Jeesmon Jacob
03/09/2023, 4:57 PM馃檪
g
Gyula F贸ra
03/09/2023, 4:58 PMMaybe this is worth a jira ticket, we could add support for standalone mode through the operator
Gyula F贸ra
03/09/2023, 4:58 PMAnd another ticket for adding this to native as well but that needs to go in Flink :)
j
Jeesmon Jacob
03/09/2023, 4:58 PMWill create two. Thanks!
g
Gyula F贸ra
03/09/2023, 4:58 PMThe standalone one is an easy fix if you want to give it a shot
j
Jeesmon Jacob
03/09/2023, 5:00 PMIs that's the right place to get number of TMs?
g
Gyula F贸ra
03/09/2023, 5:00 PMYes but we need an extra conf for the standby tm num
Gyula F贸ra
03/09/2023, 5:01 PMAnd then we can adjust the parallelism calculation
j
Jeesmon Jacob
03/09/2023, 5:01 PMso above code is not used for native mode
g
Gyula F贸ra
03/09/2023, 5:01 PMNo it's not
j
Jeesmon Jacob
03/09/2023, 5:01 PMI thought in native mode also if replicas is set for TM operator resets parallelism
g
Gyula F贸ra
03/09/2023, 5:02 PMThat's true but the JM creates tms on demand
Gyula F贸ra
03/09/2023, 5:02 PMSo it's impossible to get standbys
j
Jeesmon Jacob
03/09/2023, 5:02 PMoh ok 馃憤
k
kingsathurthi
03/10/2023, 10:27 AMis there a way to change the taskmanager parallelism value in runtime?
g
Gyula F贸ra
03/10/2023, 10:29 AMThe operator does this with standalone mode and reactive scaling enabled to scale jobs
Gyula F贸ra
03/10/2023, 10:30 AMBut reactive mode is not very stable I feel
k
kingsathurthi
03/10/2023, 10:32 AMgreat will check out, any other alternate way in flink kubernetes operator?
g
Gyula F贸ra
03/10/2023, 10:40 AMWhy do you want to do this? The operator does things for specific reasons usually :)
k
kingsathurthi
03/10/2023, 10:41 AMto do manual scaling when required without reinstalling the helmchart
g
Gyula F贸ra
03/10/2023, 1:21 PMdoesn鈥檛 really sound like a use-case we want to support through the operator. If you want to rescale a deployment, you change the spec and the operator should execute this in the most efficient way
k
kingsathurthi
03/10/2023, 1:34 PMHow I can change that through operator
g
Gyula F贸ra
03/12/2023, 1:02 PM@Jeesmon Jacob seems like this may alraedy be available in Flink actually:
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/config/#slotmanager-redundant-taskmanager-num
Gyula F贸ra
03/12/2023, 1:02 PMYou can try enabling this config for native deployments
j
Jeesmon Jacob
03/14/2023, 12:58 PMCool, let me take a look gratitude thank you
Jeesmon Jacob
03/14/2023, 1:02 PM Copy code
The number of redundant task managers. Redundant task managers are extra task managers started by Flink, in order to speed up job recovery in case of failures due to task manager lost. Note that this feature is available only to the active deployments (native K8s, Yarn).this feature is available only to the active deployments (native K8s, Yarn)mode: nativeg
Gyula F贸ra
03/14/2023, 1:31 PMyes
馃憤 1
j
Jeesmon Jacob
03/15/2023, 1:31 PMWe tried this with Flink 1.15. What we noticed is while adding redundant TMs a node scaling is triggered (which will take few minutes). As redundant TMs are not ready in some configured time, flink tries to keep adding more redundant TMs until it gets needed TMs in ready state. For each new provisioning of redundant TM job is getting restarted. Once it settled down we had 17 extra redundant TMs (request was 2) and then 15 of them removed.
Is there any config that you know can control the time interval of spinning new redundant TMs?
g
Gyula F贸ra
03/16/2023, 1:57 PMI never tried this so I know very little about it :/
j
Jeesmon Jacob
03/16/2023, 1:58 PMNo worries, thanks 馃檪
7 Views