Hi Everyone , I have an application that 1. Strea...
# troubleshooting
m
Hi Everyone , I have an application that 1. Stream data from Kafka 2. Insert the data received into Flink Table-Api 3. Perform Join on tables and emit event 4. StreamExecutionEnvironment and StreamTableEnvironment is used to execute Problem & Troubleshooting done We are trying to write Integration Test(BDD) by 1. MiniClusterWithClientResource is used to start Flink Service
Copy code
miniCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
                .setConfiguration(configuration)
                .setNumberSlotsPerTaskManager(2)
                .setNumberTaskManagers(1)
                .build());
 // Set classloader to parent-first
        configuration.setString("classloader.resolve-order", "parent-first");
PackagedProgram program = PackagedProgram.newBuilder()
                    .setArguments(args)
                    .setJarFile(new File(JAR_PATH + "APP-TO-Test.jar"))
                    .build();
            // Create Job Graph
            JobGraph jobGraph = PackagedProgramUtils.createJobGraph(program, new Configuration(), 1, true);
            assertNotNull(jobGraph, "JobGraph should be created");

            // submit a job
            miniCluster.getClusterClient().submitJob(jobGraph).get();
2. We are able to push data into Embedded kakfa 3. App is reading messages from Kafka 4. But SQL execute is not called while streaming the data , I think the reason is when streamExecutionEnvironment.execute(); is called when Flink is running inside miniCluster- StreamExecutionEnvironment._getExecutionEnvironment_() is returning StreamPlanEnvironment which have overridden executeAsync method that throws ProgramAbortException this cause the stream to be running but SQL Asynch execute method is not called due to ProgramAbortException
Copy code
@Override
    public JobClient executeAsync(StreamGraph streamGraph) {
        pipeline = streamGraph;

        // do not go on with anything now!
        throw new ProgramAbortException();
    }
Question for which i am looking for answer 1. My understanding is here instead if call to StreamExecutionEnvironment._getExecutionEnvironment_() inside miniCluster would have returned LocalStreamEnvironment then SQL execution would have performed as part of executeAsync ( when App is executed inside in Apache Flink then StreamExecutionEnvironment._getExecutionEnvironment_() is returning LocalStreamEnvironment) 2. Who can i make sure SQL execute call works while app is running inside miniCluster( during Integration Test) @David Anderson
d
m
Thank you will check it.
k
Hi @David Anderson, I have tried the above approach and I can get the sql to run in my application. Thanks for your suggestion. However, when I use this to send 1 message, it works fine. When I publish 11 messages, I get :
Caused by: java.io.IOException: Insufficient number of network buffers: required 13, but only 7 available. The total number of network buffers is currently set to 2048 of 32768 bytes each. You can increase this number by setting the configuration keys 'taskmanager.memory.network.fraction', 'taskmanager.memory.network.min', and 'taskmanager.memory.network.max'.
I have increased the memory, still not able resolve this. Configuration is as follows:
Copy code
Configuration configuration = new Configuration();
configuration.setString(String.valueOf(TaskManagerOptions.MANAGED_MEMORY_SIZE), "512m");
configuration.setString("taskmanager.memory.task.off-heap.size", "256m");
configuration.setString("taskmanager.memory.framework.off-heap.size", "128m");
configuration.setString("taskmanager.memory.network.min", "64m");
configuration.setString("taskmanager.memory.network.max", "512m");
configuration.setString("classloader.resolve-order", "parent-first");
configuration.setString("restart-strategy", "fixed-delay");
configuration.setString("restart-strategy.fixed-delay.attempts", "3");
configuration.setString("restart-strategy.fixed-delay.delay", "60s");
configuration.setString("taskmanager.network.memory.buffer-debloat.enabled", "true");
d
Did you try without any taskmanager memory settings? I would expect the defaults to work.
k
sure. I will try that
Copy code
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
          	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
          	at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
          	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
          	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
          	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
          	at org.apache.flink.runtime.rpc.pekko.PekkoInvocationHandler.lambda$invokeRpc$1(PekkoInvocationHandler.java:268)
          	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
          	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
          	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
          	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
          	at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1287)
          	at org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
          	at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
          	at org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
          	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
          	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
          	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
          	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
          	at org.apache.flink.runtime.concurrent.pekko.ScalaFutureUtils$1.onComplete(ScalaFutureUtils.java:47)
          	at org.apache.pekko.dispatch.OnComplete.internal(Future.scala:310)
          	at org.apache.pekko.dispatch.OnComplete.internal(Future.scala:307)
          	at org.apache.pekko.dispatch.japi$CallbackBridge.apply(Future.scala:234)
          	at org.apache.pekko.dispatch.japi$CallbackBridge.apply(Future.scala:231)
          	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
          	at org.apache.flink.runtime.concurrent.pekko.ScalaFutureUtils$DirectExecutionContext.execute(ScalaFutureUtils.java:65)
          	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
          	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
          	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
          	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
          	at org.apache.pekko.pattern.PromiseActorRef.$bang(AskSupport.scala:624)
          	at org.apache.pekko.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:33)
          	at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:536)
          	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
          	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
          	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
          	at org.apache.pekko.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:73)
          	at org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:110)
          	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
          	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
          	at org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:110)
          	at org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59)
          	at org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:61)
          	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
          	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
          	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
          	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
          	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
          Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
          	at org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:219)
          	at org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailureAndReport(ExecutionFailureHandler.java:166)
          	at org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:121)
          	at org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:281)
          	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:272)
          	at org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:265)
          	at org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:787)
          	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:764)
          	at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83)
          	at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:515)
          	at jdk.internal.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
          	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
          Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy

          	at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:318)
          	at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
          	at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:316)
          	at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:229)
          	at org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:88)
          	at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:174)
          	at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
          	at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
          	at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
          	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
          	at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
          	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
          	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
          	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
          	at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
          	at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
          	at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
          	at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
          	at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
          	at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:272)
          	at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:233)
          	at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:245)
          	... 5 more
          Caused by: java.io.IOException: Insufficient number of network buffers: required 13, but only 7 available. The total number of network buffers is currently set to 2048 of 32768 bytes each. You can increase this number by setting the configuration keys 'taskmanager.memory.network.fraction', 'taskmanager.memory.network.min', and 'taskmanager.memory.network.max'.
          	at org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.internalCreateBufferPool(NetworkBufferPool.java:494)
          	at org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.createBufferPool(NetworkBufferPool.java:467)
          	at org.apache.flink.runtime.io.network.partition.ResultPartitionFactory.lambda$createBufferPoolFactory$0(ResultPartitionFactory.java:399)
          	at org.apache.flink.runtime.io.network.partition.ResultPartition.setup(ResultPartition.java:158)
          	at org.apache.flink.runtime.taskmanager.Task.setupPartitionsAndGates(Task.java:981)
          	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:669)
          	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
          	at java.base/java.lang.Thread.run(Thread.java:829)
I get the above when I try without taskmanager memory settings
Hi @David Anderson , Thank you very much for your help. This issue is solved and we got the BDDs to run.
m
We had to play around "taskmanager.memory.network.fraction" to fix the memory issue "taskmanager.memory.network.fraction", "0.01" Thank you for your inputs and help @David Anderson
πŸ‘ 1