Zhong Chen
09/12/2023, 9:42 PMpipeline.max-parallelism
can’t be updated? Getting the below error when updating the setting for my job.
Caused by: java.util.concurrent.CompletionException: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint Checkpoint Metadata. Max parallelism mismatch between checkpoint/savepoint state and new program. Cannot map operator 77b992fbfa92ed1320e4d6cad773ae9f with max parallelism 128 to new program with max parallelism 256. This indicates that the program has been changed in a non-compatible way after the checkpoint/savepoint.
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint Checkpoint Metadata. Max parallelism mismatch between checkpoint/savepoint state and new program. Cannot map operator 77b992fbfa92ed1320e4d6cad773ae9f with max parallelism 128 to new program with max parallelism 256. This indicates that the program has been changed in a non-compatible way after the checkpoint/savepoint.
at org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:190) ~[flink-dist-1.17.1.jar:1.17.1]
Martijn Visser
09/13/2023, 7:34 AMThere is currently no way to change the maximum parallelism of an operator after a job has started without discarding that operators state.
David Anderson
09/13/2023, 3:54 PMZhong Chen
09/13/2023, 4:20 PMDavid Anderson
09/13/2023, 11:17 PM