Hello,
I get following events when a job manager i...
# troubleshootings
Sumit Nekar
01/24/2023, 5:46 PMHello,
I get following events when a job manager is starting during upgrade and job manager gets restarted after these events and it comes up fine.
I suspect this is something related leader election process. Please suggest if you have any inputs.
Job is being deployed using flink kuberenetes operator.
}
2023-01-24 17:43:25,532 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job 7eef2349fa26a94509d5126440972a25 is submitted.2023-01-24 17:43:25,532 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=7eef2349fa26a94509d5126440972a25.2023-01-24 17:43:27,040 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.2023-01-24 17:43:27,042 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting KubernetesApplicationClusterEntrypoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..2023-01-24 17:43:27,044 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting down rest endpoint.2023-01-24 17:43:27,299 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon.2023-01-24 17:43:27,299 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon.2023-01-24 17:43:27,301 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports.2023-01-24 17:43:27,301 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports.2023-01-24 17:43:27,313 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down.2023-01-24 17:43:27,313 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down.g
Guruguha Marur Sreenivasa
01/24/2023, 6:23 PMIf you're running on session mode, does your job manager restart at this time? If so, check the meta space memory allocated to it.
s
Sumit Nekar
01/25/2023, 3:40 AMJob is deployed in application mode using flink kubernetes operator
Sumit Nekar
01/25/2023, 3:40 AMJob manager restarts even before it spawns up TM pods
w
weijie guo
01/31/2023, 6:06 AMHave you checked whether JM is out of memory
s
Sumit Nekar
01/31/2023, 6:40 AMIt doesnt seem so. It has enough memory as per metrics.