I'm new to Flink... I'm using flink SQL. I use sql...
# troubleshooting
y
I'm new to Flink... I'm using flink SQL. I use sql-client to run my sql jobs. Set parallelism.default=1. And set my sink table to sink.parallelism = 4 When the job starts, the GlobalWindowAggregate task parallelism is 1, I can only set it to 4 using the UI (API). Anyways that can be set at the SQL level? My issue, is that when I set all to 4, data is coming in faster that it can be processed, and eventually the job fails. Thank you
p
Hello there Younes! Which Flink version are you using? Flink 1.19 added FLIP-367 which add in the configurations
scan.parallelism
I this this could help you https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sourcessinks/#scan-table-source
y
I'm using 1.20.1
p
So this option should be available. Have you tried it?
y
Nope, will do now 🙂
🙌 1
Thank you
p
No worries 😄
y
It doesn't seem to have an affect. I ran this
Copy code
SET 'parallelism.default' = '1';
SET 'scan.parallelism' = '4';
And my sink table has
Copy code
'sink.parallelism' = '4'
And tasks look like
Other than UI, I can't seem to be able to increase tasks for
GlobalWindowAggregate
Am I missing somehting?
p
This should be set on the create table 😄
y
AH, crap... will do now 🙂
Copy code
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.ValidationException: Unsupported options found for 'kafka'.

Unsupported options:

scan.parallelism
these are the only scan option for Kafka
Copy code
scan.bounded.mode
scan.bounded.specific-offsets
scan.bounded.timestamp-millis
scan.startup.mode
scan.startup.specific-offsets
scan.startup.timestamp-millis
scan.topic-partition-discovery.interval
scan.watermark.alignment.group
scan.watermark.alignment.max-drift
scan.watermark.alignment.update-interval
scan.watermark.emit.strategy
scan.watermark.idle-timeout
p
Hum. So it looks like
datagen
connector supports it. But not
kafka
. As far as I know, besides this option, there is no way to control fine grained parallelism on SQL/Table API. I think only datastream api would give you this option
y
Thank you for the help 🙂
The basic issue, is that I get more data that I can process...
And this only an issue when backfilling
On normal oprtation, I'm fine
p
No worries. It looks like implemented but not merged. See here. You could headsup #dev or create a fork with this PR merged.
gratitude thank you 1
y
So, if I can trohtle the Kafka reading, I should be fine
p
Perhaps try the other way around. Set the general parallelism, and on the sink connector use
sink.parallelism = 4
perhaps it may do the trick 😅
y
Tried it, it didn't... The blockage seems to be at the transformation
😓 1
So, I merged a the in a private build, and voila:
Thanks for your help! Really appreciate it
p
Nice! Glad it helped
y
My last one
🔥 1
c
@Younes Naguib Are you running your Gateway on Kubernetes? I just started a thread looking for info right below your thread. I could use some basic info.
y
Yes, running in k8s
p
y
Very nice 🙂 Thank you
So, just a matter of waiting for the next release
🙌 1