Hello!
I’m testing a Autoscaler in Flink Kubernete...
# troubleshootingt
Tan Kim
04/27/2023, 1:05 AMHello!
I’m testing a Autoscaler in Flink Kubernetes Operator v1.4.0 these days, there is some issue.
When some job is scaled down, the error is occurred. (detail error in thread)
If there is anyone who resolve this, please let me share, thank you!
Flink: 1.17.0
FKO: 1.4.0
StateBackend: RocksDB(GIC & UC enabled)
Tan Kim
04/27/2023, 1:07 AM Copy code
java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5
at org.apache.flink.runtime.io.network.partition.PipelinedResultPartition.getCheckpointedSubpartition(PipelinedResultPartition.java:183)
at org.apache.flink.runtime.checkpoint.channel.ResultSubpartitionRecoveredStateHandler.getSubpartition(RecoveredChannelStateHandler.java:222)
at org.apache.flink.runtime.checkpoint.channel.ResultSubpartitionRecoveredStateHandler.lambda$calculateMapping$1(RecoveredChannelStateHandler.java:237)
at java.base/java.util.stream.IntPipeline$1$1.accept(Unknown Source)
at java.base/java.util.Spliterators$IntArraySpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.Spliterator$OfInt.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
at org.apache.flink.runtime.checkpoint.channel.ResultSubpartitionRecoveredStateHandler.calculateMapping(RecoveredChannelStateHandler.java:238)
at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
at org.apache.flink.runtime.checkpoint.channel.ResultSubpartitionRecoveredStateHandler.getMappedChannels(RecoveredChannelStateHandler.java:227)
at org.apache.flink.runtime.checkpoint.channel.ResultSubpartitionRecoveredStateHandler.getBuffer(RecoveredChannelStateHandler.java:182)
at org.apache.flink.runtime.checkpoint.channel.ResultSubpartitionRecoveredStateHandler.getBuffer(RecoveredChannelStateHandler.java:157)
at org.apache.flink.runtime.checkpoint.channel.ChannelStateChunkReader.readChunk(SequentialChannelStateReaderImpl.java:198)
at org.apache.flink.runtime.checkpoint.channel.SequentialChannelStateReaderImpl.readSequentially(SequentialChannelStateReaderImpl.java:107)
at org.apache.flink.runtime.checkpoint.channel.SequentialChannelStateReaderImpl.read(SequentialChannelStateReaderImpl.java:93)
at org.apache.flink.runtime.checkpoint.channel.SequentialChannelStateReaderImpl.readOutputData(SequentialChannelStateReaderImpl.java:79)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:731)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.base/java.lang.Thread.run(Unknown Source)g
Gyula Fóra
04/27/2023, 11:07 AMSo the error seems to happen when scaling down the job?
Gyula Fóra
04/27/2023, 11:07 AMDoes the same error also happen if for the same checkpoint you manually change the entire job parallelism?
t
Tan Kim
04/27/2023, 11:47 AMSo the error seems to happen when scaling down the job?Yes, when the parallelism scaled down.
Does the same error also happen if for the same checkpoint you manually change the entire job parallelism?I’ve not been changed it manually from that checkpoint. Why is that needed? Is there any difference between those?
g
Gyula Fóra
04/27/2023, 11:49 AMI wouldn’t expect a difference. But in any case this seems to be an error with checkpoint restore in general
Gyula Fóra
04/27/2023, 11:49 AMnot related to the autoscaler itself
Gyula Fóra
04/27/2023, 11:50 AMCould you please open a jira ticket with the details and any extra JM/TM logs you may have?
Gyula Fóra
04/27/2023, 11:50 AMWhat is GIC & UC?
t
Tan Kim
04/27/2023, 11:51 AMGeneric Incremental Checkpoint and Unalignment Checkpoint, it’s mentioned here.
g
Gyula Fóra
04/27/2023, 11:52 AMok, so the bug may be related to those components. We generally use this with aligned checkpoints
👀 1
Gyula Fóra
04/27/2023, 11:52 AMand I haven’t seen issue like this with that
t
Tan Kim
04/27/2023, 11:52 AMI’ve never been open jira ticket before, could you please let me know how should I open?
g
Gyula Fóra
04/27/2023, 11:54 AMYou can look at other jira tickets as examples, such as: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-30719
Gyula Fóra
04/27/2023, 11:54 AMyou create a new ticket, set the correct component, flink version. Provide all the info and logs if possible
🙆 1
t
Tan Kim
04/27/2023, 11:54 AMok, so the bug may be related to those components. We generally use this with aligned checkpointsThat’s why you asked if I’ve tried restoring from a checkpoint manually.
Tan Kim
04/28/2023, 2:01 AM4 Views