Younes Naguib
02/20/2025, 4:19 PMPedro Mázala
02/20/2025, 4:29 PMscan.parallelism
I this this could help you https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sourcessinks/#scan-table-sourceYounes Naguib
02/20/2025, 4:29 PMPedro Mázala
02/20/2025, 4:29 PMYounes Naguib
02/20/2025, 4:30 PMYounes Naguib
02/20/2025, 4:30 PMPedro Mázala
02/20/2025, 4:30 PMYounes Naguib
02/20/2025, 4:41 PMSET 'parallelism.default' = '1';
SET 'scan.parallelism' = '4';
And my sink table has
'sink.parallelism' = '4'
And tasks look likeYounes Naguib
02/20/2025, 4:41 PMGlobalWindowAggregateYounes Naguib
02/20/2025, 4:41 PMPedro Mázala
02/20/2025, 4:41 PMYounes Naguib
02/20/2025, 4:42 PMYounes Naguib
02/20/2025, 4:43 PM[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.ValidationException: Unsupported options found for 'kafka'.
Unsupported options:
scan.parallelismYounes Naguib
02/20/2025, 4:44 PMscan.bounded.mode
scan.bounded.specific-offsets
scan.bounded.timestamp-millis
scan.startup.mode
scan.startup.specific-offsets
scan.startup.timestamp-millis
scan.topic-partition-discovery.interval
scan.watermark.alignment.group
scan.watermark.alignment.max-drift
scan.watermark.alignment.update-interval
scan.watermark.emit.strategy
scan.watermark.idle-timeoutPedro Mázala
02/20/2025, 4:50 PMdatagen connector supports it. But not kafka. As far as I know, besides this option, there is no way to control fine grained parallelism on SQL/Table API.
I think only datastream api would give you this optionYounes Naguib
02/20/2025, 4:50 PMYounes Naguib
02/20/2025, 4:51 PMYounes Naguib
02/20/2025, 4:52 PMYounes Naguib
02/20/2025, 4:52 PMPedro Mázala
02/20/2025, 4:52 PMYounes Naguib
02/20/2025, 4:52 PMPedro Mázala
02/20/2025, 4:54 PMsink.parallelism = 4Pedro Mázala
02/20/2025, 4:54 PMYounes Naguib
02/20/2025, 4:55 PMYounes Naguib
02/20/2025, 6:29 PMYounes Naguib
02/20/2025, 6:29 PMPedro Mázala
02/20/2025, 6:31 PMYounes Naguib
02/20/2025, 6:46 PMChad Theis
02/20/2025, 11:47 PMYounes Naguib
02/21/2025, 7:51 PMPedro Mázala
03/18/2025, 10:58 AMYounes Naguib
03/18/2025, 12:41 PMYounes Naguib
03/18/2025, 12:41 PM