Francis Altomare
06/02/2026, 2:43 PMupgradeMode last-state and using the adaptive scheduler we observe:
• Downscale rescales (e.g. parallelism 24 -> 12) take roughly 2 hours from spec change to RUNNING (no events are processed in this time
• Upscale rescales in the same flow are much faster, taking around 12 minutes
The relevant ForSt / checkpointing settings are:
• use-ingest-db-restore-mode: true
• incremental-restore-async-compact-after-rescale: true
• state-recovery.claim-mode: CLAIM
• NATIVE savepoint format, incremental + compressed checkpoints
• pipeline.max-parallelism: 120
• ForSt cache size-based-limit 350GB on a 500Gi gp3 volume per TM.
• I’m running one slot per TM and have tried parallelisms between 12 and 24
My questions are:
• Is ~2h for a downscale at this state size in line with what others see, or does it suggest a misconfiguration? Most of the time appears to be in restore.
• Is the asymmetry where upscales are much faster also expected?
• Given the asymmetry, are there settings or patterns that make autoscaling viable at ~1TB state beyond adjusting job.autoscaler.scale-down.max-factor and scale-down.interval? Or is the practical recommendation at this scale to keep parallelism static and rescale manually during low-traffic windows?
Thank you!Bianca Falcone
06/03/2026, 8:48 AMDanny Wilkins
06/08/2026, 3:52 PMpekko tests try to connect to my machine's local-but-external ip (ie 10.0.0.whatever.) Is there a property or something else I can just -D in order to force pekko to use localhost instead of whatever discovery mechanism it's doing?HunkSurvivor89
06/08/2026, 6:07 PMHan You
06/23/2026, 6:08 PMOr Keren
07/01/2026, 12:30 PMAsyncException{org.forstdb.RocksDBException: Exception when Flush file, path: <s3://yotpo-cdp/flinkcdp/checkpoints/shared/op_AsyncKeyedProcessOperator_f199ad3ac3e86e5fc9372ebb239a06cb__33_64__attempt_0/db/260675.sst>}
at org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncKeyOrderedStreamOperator.handleAsyncException(AbstractAsyncKeyOrderedStreamOperator.java:120)
at org.apache.flink.core.asyncprocessing.AsyncFutureImpl.completeExceptionally(AsyncFutureImpl.java:553)
at org.apache.flink.state.forst.ForStDBPutRequest.completeStateFutureExceptionally(ForStDBPutRequest.java:87)
at org.apache.flink.state.forst.ForStWriteBatchOperation.lambda$process$0(ForStWriteBatchOperation.java:80)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)
at org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:247)
at java.base/java.util.concurrent.CompletableFuture.asyncRunStage(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.runAsync(Unknown Source)
at org.apache.flink.state.forst.ForStWriteBatchOperation.process(ForStWriteBatchOperation.java:65)
at org.apache.flink.state.forst.ForStStateExecutor.lambda$executeBatchRequests$2(ForStStateExecutor.java:201)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.flink.state.forst.ForStStateExecutor$ForStExecutorThreadFactory.lambda$newThread$0(ForStStateExecutor.java:338)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.forstdb.RocksDBException: Exception when Flush file, path: <s3://yotpo-cdp/flinkcdp/checkpoints/shared/op_AsyncKeyedProcessOperator_f199ad3ac3e86e5fc9372ebb239a06cb__33_64__attempt_0/db/260675.sst>
at org.forstdb.RocksDB.write0(Native Method)
at org.forstdb.RocksDB.write(RocksDB.java:1843)
at org.apache.flink.state.forst.ForStDBWriteBatchWrapper.flush(ForStDBWriteBatchWrapper.java:136)
at org.apache.flink.state.forst.ForStDBWriteBatchWrapper.flushIfNeeded(ForStDBWriteBatchWrapper.java:155)
at org.apache.flink.state.forst.ForStDBWriteBatchWrapper.put(ForStDBWriteBatchWrapper.java:117)
at org.apache.flink.state.forst.ForStDBPutRequest.process(ForStDBPutRequest.java:68)
at org.apache.flink.state.forst.ForStWriteBatchOperation.lambda$process$0(ForStWriteBatchOperation.java:70)
... 10 more
Suppressed: org.forstdb.RocksDBException: Exception when Flush file, path: <s3://yotpo-cdp/flinkcdp/checkpoints/shared/op_AsyncKeyedProcessOperator_f199ad3ac3e86e5fc9372ebb239a06cb__33_64__attempt_0/db/260675.sst>
at org.forstdb.RocksDB.write0(Native Method)
at org.forstdb.RocksDB.write(RocksDB.java:1843)
at org.apache.flink.state.forst.ForStDBWriteBatchWrapper.flush(ForStDBWriteBatchWrapper.java:136)
at org.apache.flink.state.forst.ForStDBWriteBatchWrapper.close(ForStDBWriteBatchWrapper.java:144)
at org.apache.flink.state.forst.ForStWriteBatchOperation.lambda$process$0(ForStWriteBatchOperation.java:67)
... 10 moreFabrizzio Chavez
07/03/2026, 1:13 AMSơn Bùi
07/08/2026, 4:09 AMKaran Kumar
07/09/2026, 5:25 AMDavid Causse
07/10/2026, 2:07 PMcompatibleAfterMigration. Initially my understanding was that the old serializer is used to read the stored bytes, the instance object is then written with the new serializer and these new bytes are read by the new serializer, in short I would have assumed that it ensured that the instances received in the operators are always constructed using the new serializer. But I'm hitting a bug where I suspect that the instance read by the old serializer is simply passed to the operator without any migration. (Note this is the operator state of the Async operator). Could someone clarify my understanding about state migration with compatibleAfterMigration? Thanks! 🙏pavithranagayallapu
07/17/2026, 11:12 AMMaitreya Manohar
07/20/2026, 7:05 AMpavithranagayallapu
07/20/2026, 11:05 AMzhongxia.zhou
07/20/2026, 9:20 PMFrancis Altomare
07/22/2026, 8:22 AMpavithranagayallapu
07/23/2026, 3:31 AMpavithranagayallapu
07/23/2026, 3:43 AMSiddhesh Kalgaonkar
08/04/2026, 3:43 PMUdith V B
08/06/2026, 1:21 PMNishtha Bhattacharjee
08/09/2026, 9:19 PMnoa.cavassi
08/12/2026, 12:20 PM[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.iceberg.data.IcebergGenericReader
... 15 more
And the full error in the jobmanager out file:
java.lang.RuntimeException: One or more fetchers have encountered exception
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:333) ~[flink-connector-files-1.20.0.jar:1.20.0]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:228) ~[flink-connector-files-1.20.0.jar:1.20.0]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:190) ~[flink-connector-files-1.20.0.jar:1.20.0]
at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:422) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) ~[flink-dist-1.20.0.jar:1.20.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-dist-1.20.0.jar:1.20.0]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.lang.NoClassDefFoundError: org/apache/iceberg/data/IcebergGenericReader
at org.apache.fluss.lake.iceberg.source.IcebergRecordReader.<init>(IcebergRecordReader.java:65) ~[fluss-lake-iceberg-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.lake.iceberg.source.IcebergLakeSource.createRecordReader(IcebergLakeSource.java:100) ~[fluss-lake-iceberg-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.lake.reader.LakeSnapshotScanner.pollBatch(LakeSnapshotScanner.java:54) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.source.reader.BoundedSplitReader.pollBatch(BoundedSplitReader.java:128) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.source.reader.BoundedSplitReader.poll(BoundedSplitReader.java:121) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.source.reader.BoundedSplitReader.readBatch(BoundedSplitReader.java:76) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.source.reader.FlinkSourceSplitReader.fetch(FlinkSourceSplitReader.java:154) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) ~[flink-connector-files-1.20.0.jar:1.20.0]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165) ~[flink-connector-files-1.20.0.jar:1.20.0]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:117) ~[flink-connector-files-1.20.0.jar:1.20.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
... 1 more
Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.data.IcebergGenericReader
at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
at org.apache.fluss.lake.iceberg.source.IcebergRecordReader.<init>(IcebergRecordReader.java:65) ~[fluss-lake-iceberg-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.lake.iceberg.source.IcebergLakeSource.createRecordReader(IcebergLakeSource.java:100) ~[fluss-lake-iceberg-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.lake.reader.LakeSnapshotScanner.pollBatch(LakeSnapshotScanner.java:54) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.source.reader.BoundedSplitReader.pollBatch(BoundedSplitReader.java:128) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.source.reader.BoundedSplitReader.poll(BoundedSplitReader.java:121) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.source.reader.BoundedSplitReader.readBatch(BoundedSplitReader.java:76) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.fluss.flink.source.reader.FlinkSourceSplitReader.fetch(FlinkSourceSplitReader.java:154) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) ~[flink-connector-files-1.20.0.jar:1.20.0]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165) ~[flink-connector-files-1.20.0.jar:1.20.0]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:117) ~[flink-connector-files-1.20.0.jar:1.20.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
... 1 more
Any ideas on what the problem could be? I'm happy to provide any other information if necessary.Gwenael LE BARZIC
08/31/2026, 3:40 PMIrene Rodriguez
09/02/2026, 7:56 AMSourceSplitMetricGroup), but these metric groups are never released after the split finishes. Over a few days, this fills up the heap, triggers heavy GC saturation, causes source starvation, and increases consumer lag until we restart the job.
Our heap dump shows that 99.9% of the leaked metric groups contain the 'split' key.
Is there already a JIRA issue or a PR tracking this lifecycle cleanup bug for FLIP-513? If not, we can create a JIRA issue with our heap dump findings.
Also, is there a recommended configuration workaround to completely disable/exclude these split-level watermark metrics in Flink 2.2 to prevent registration in the meantime?
Thanks!Tino Lyu
09/02/2026, 4:48 PMFutureCompletingBlockingQueue.waitOnPut() that permanently stalls a fetcher thread under sustained backpressure after a mass split-assignment burst.
The ticket shows this is resolved upstream, with fix versions including 1.20.6, 2.0.3, 2.1.4, 2.2.2, 2.3.0, and 2.4.0. Only 2.3.0 has actually shipped so far.
A couple of questions for the community:
1. Is there a workaround for this that doesn't require a full Flink version bump?
2. Is there a rough timeline for a 1.20.6 patch release, or is upgrading to 2.3.0 the recommended path forward?
Thanks in advance.Brandon Andersen
09/04/2026, 1:58 PMJdbcSink from the JDBC Connector. Can someone take a look at it for me? Happy to make any changes to it too. github.com/apache/flink/pull/27528vishal.balaraman
09/07/2026, 9:34 AMflink-cdc-pipeline-connector-iceberg (3.6.0-2.2) to stream MySQL → Iceberg on Glue.
When the partition key column is updated, we’re getting duplicate rows instead of the row being moved to the new partition.
Repro:
CREATE TABLE t (
id BIGINT,
val VARCHAR,
region VARCHAR
) PARTITIONED BY identity(region);
INSERT id=1, region=north;
UPDATE id=1 SET region=east;
After the update, id=1 appears twice: one row with region=north (old/stale) and one with region=east (new).
From what I understand, this happens because the equality delete only uses the primary key and doesn’t include the partition column, so the delete doesn’t remove the row from the old partition.
Has anyone solved this for partitioned tables? Is scheduled compaction the only practical workaround, or is there a write-side fix?
Also, does flink-cdc-pipeline-connector-iceberg expose any configuration to set the equality fields?
Any pointers would be appreciated.Fabricio Lemos
09/16/2026, 9:15 PMdrzymalamateusz
09/18/2026, 8:48 AMAvi Sanwal
09/22/2026, 3:24 PMumar farooq
09/23/2026, 4:16 AM