Hi All, during flink job upgrade, sometimes we obs...
# troubleshooting
a
Hi All, during flink job upgrade, sometimes we observe an exception like (attached the full stacktrace):
Copy code
org.apache.flink.runtime.resourcemanager.exceptions.UnknownTaskExecutorException: No TaskExecutor registered under my-flink-app-taskmanager-1-27
Has anyone encountered this before? Do you know how we can fix this? We are using flink 1.17.0 hosted on EKS and managed using flink-kubernetes-operator version 1.4.0. The flink job is deployed in application mode (FlinkDeployment CRD).
d
is there a network partition between the TaskManagers and Resource maangers?
Make sure the task manager pod in question is up and running.
a
@D. Draco O'Brien, the flink-kubernetes-operator manages the pods for us. The job managers, task managers and operator are all hosted on the same EKS cluster. My suspicion right now is that during upgrade, the operator deletes older task managers asyncronously and starts deploying the new job manager. On a race condition, older task managers which are yet to be deleted may be talking to the new job manager which does not recognize it.
d
only the logs will tell!
πŸ‘€ 1