Hello All , We have two flink pipelines 1) 1 job...
# troubleshooting
j
Hello All , We have two flink pipelines 1) 1 job manager and 5 task manager 2) 3 job managers and 5 task manager We are using flink:1.14.4-scala_2.12-java11 and lyft flinkk8soperator We have deployed this flink on a multi host k8s cluster In both the above pipelines we see the task managers restarting with below exceptions
Copy code
2023-07-14 10:56:56.204 [flink-akka.actor.default-dispatcher-17] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: Custom Source -> Map (7/15) (73095e8cac2266887f4b7a0fc109c199) switched from DEPLOYING to FAILED on 10.233.118.126:43710-2c7cb4 @ 10.233.118.126 (dataPort=34133).
org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException: Could not send message [RemoteRpcInvocation(null.submitTask(TaskDeploymentDescriptor, JobMasterId, Time))] from sender [Actor[<akka://flink/temp/taskmanager_0$+rod]]> to recipient [Actor[<akka.tcp://flink@10.233.118.126:43710/user/rpc/taskmanager_0#-1148004411]]>, because the recipient is unreachable. This can either mean that the recipient has been terminated or that the remote RpcService is currently not reachable.
        at org.apache.flink.runtime.rpc.akka.DeadLettersActor.handleDeadLetter(DeadLettersActor.java:61)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
        at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
        at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
        at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
        at akka.actor.Actor.aroundReceive(Actor.scala:537)
        at akka.actor.Actor.aroundReceive$(Actor.scala:535)
        at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
        at akka.actor.ActorCell.invoke(ActorCell.scala:548)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
m
The Lyft operator isn’t maintained by the Flink community
So I think it will be hard to help with this problem. I would recommend using the operator from Flink itself