I read a bunch of data from Kafka in a batch job, ...
# troubleshooting
j
I read a bunch of data from Kafka in a batch job, keyBy and in the next operator downstream, the data is aggregated. I don't have enough disk space to write all the raw data to disk. I need the task before the shuffle with the Kafka Source and the task after the shuffle with the aggregation to run at the same time, so I don't have to spill all the raw data to disk. From what I read in the documentation this should be possible with execution.batch-shuffle-mode=ALL_EXCHANGES_HYBRID_SELECTIVE I run the task like this: flink run -Dexecution.batch-shuffle-mode=ALL_EXCHANGES_HYBRID_SELECTIVE my.jar But in the flink UI I can see that only the kafka source starts. The task with the aggregation does not start. I attached a screenshot.
okay, I had to add -Djobmanager.scheduler=Default. Now it works
What does this mean, I cannot find anything about this error.
Copy code
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Error at remote task manager '<redacted> [ <redacted> ] '.
	at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.decodeMsg(CreditBasedPartitionRequestClientHandler.java:327)
	at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelRead(CreditBasedPartitionRequestClientHandler.java:191)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelRead(NettyMessageClientDecoderDelegate.java:112)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
	at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
	at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Result partition has been already released.
	at org.apache.flink.runtime.io.network.partition.hybrid.HsFileDataManager.release(HsFileDataManager.java:208)
	at org.apache.flink.runtime.io.network.partition.hybrid.HsResultPartition.releaseInternal(HsResultPartition.java:283)
	at org.apache.flink.runtime.io.network.partition.ResultPartition.release(ResultPartition.java:272)
	at org.apache.flink.runtime.io.network.partition.ResultPartitionManager.releasePartition(ResultPartitionManager.java:180)
	at org.apache.flink.runtime.io.network.partition.ResultPartition.fail(ResultPartition.java:294)
	at org.apache.flink.runtime.taskmanager.Task.failAllResultPartitions(Task.java:1009)
	at org.apache.flink.runtime.taskmanager.Task.releaseResources(Task.java:995)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:843)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
	... 1 more
I ran out of disk on one of the taskmanagers.