https://flink.apache.org/ logo
Join Slack
Powered by
I am expermenting with flink k8s operator and tr...
# random
s
I am expermenting with flink k8s operator and trying to understand how to scale my flink job. I can see that we can change the overall parallelism of a job in the ‘job’ section of FlinkDeployment. Is there any option to change only a subtask’s/operator’s parallelism ?
g
The Flink Operator only controls the overall (default) job parallelism. So this will work best if you don’t set specific operator parallelism (or you set it relative to it)
It would be impossible to control parallelism on the operator level, if the user has overrides for it and we are not planning to go down this route
Could you please explain your use-case and why you think this would be good for you?
s
I have a job with a source -> map -> processFunction->Sink In processFunction I need to connect to external db and do some processing. In this case i would want to set lower parallelism to source-map operators and higher parallelism to processFunction
In case I want to increase the parallelism of processFunction in later stages after my job is deployed in production
I think only option is to rebuild my jar with new operator parallelism and set the overall parallelism
g
What you could try is to keep the processFunction parallelism on default, and set the source/sink to specific values from config (or get parallelism from env and set it as a percentage of max parallelism)
this way you can cap the parallelism of your source/sink and your processFunction will scale according to the parallelism you set through the operator CR
s
Can we set as a percentage of default parallelism
?
g
Actually that might not be possible, you are right…
In any case you could pass your source/sink parallelism as arguments to your job
👍 1
Would it hurt your job to have the same high parallelism for all operators?
s
Just want to have better resource utilisation
How can we pass arguments in FlinkDeployment
g
s
Had checked this I guess. I din find any such example
Copy code
jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 2
    upgradeMode: stateless
    state: running
i can see these
may be flinkConfiguration?
g
Copy code
job:
    jarURI: local:///opt/flink/usrlib/sql-runner.jar
    args: ["/opt/flink/usrlib/sql-scripts/simple.sql"]
    parallelism: 1
    upgradeMode: stateless
there is an args field in the job spec
you can either pass a config file here that you packaged or your custom config values directly
it’s completely flexible, please try it
s
Sure thanks
if i make any changes in ‘Job’ section of FlinkDeployment, operator would redeploy my job right?
j
Hi @Gyula Fóra @Sumit Nekar How can we set operatorParallelism for each FlinkSessionJob? I see only job level parallelism from CRD https://github.com/apache/flink-kubernetes-operator/blob/main/helm/flink-kubernetes-operator/crds/flinksessionjobs.flink.apache.org-v1.yml
g
@Jashwanth S J please do not tag people directly on these questions. You can simply ask them in the troubleshooting channel if you can’t figure out the answer from the Flink / Operator config docs.
j
Sure @Gyula Fóra
m
I've deleted the direct ping as well. Like Gyula said, please don't directly ping people
✅ 1
4 Views
Open in Slack
PreviousNext
Powered by