Hi, I am running a streaming job in application mo...
# troubleshooting
f
Hi, I am running a streaming job in application mode that uses reactive scheduler, running on k8s with horizontal pod autoscaler for the task managers The job is quite simple: (source)->(map)->(sink) it's CPU bound in the map step currently the parallelism is set to numOfTaskManagers * cpusPerTaskManager, e.g. there are 32 task managers, 8 CPUs per TM, the parallelism is 256 this works ok, but with growing parallelism the high number of sources and sinks create issues on our infra I would like to achieve • the behaviour of reactive scheduler (automatic rescale of the job as TM pods are added / removed) • somehow modify the parallelism of source and sinks to some multiple of numOfTaskManagers (with reactive mode the parallelism is ignored and it changes as task managers are added / removed) e.g when running with 4 task managers I want 8 (= 4 * 2) source subtasks, 32 =(4*8) map subtasks and 8 (=4*2) sink subtasks with 32 task managers it would be source 64 (=32 * 2), map 256(= 32*8), sink 64=(32*2), all evenly spread out across task managers what's the best way to achieve that?
p
this works ok, but with growing parallelism the high number of sources and sinks create issues on our infra
You can define parallelism per operator in flink. Then you fix max parallelism to source and sink. This way you won't suffer from this and will be able to scale processing.
f
the per operator parallelism is ignored in reactive mode of the adaptive scheduler
We are using this https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/elastic_scaling/#reactive-mode
The parallelism of individual operators in a job will be determined by the scheduler. It is not configurable and will be ignored if explicitly set, either on individual operators or the entire job.
we are ok to abandon this, if we can get the similar behaviour with scaling up and down with number of pods
b
create a statement to wrap them into a <PARALLELISTM> task slot
i have 200 topics of source and sink, i just use 4 task slot to make it done.
p
the per operator parallelism is ignored in reactive mode of the adaptive scheduler
Parallelism yes. But not
maxParallelism
no. I think this could be your trick.
f
that makes savepoints incompatible, and is not dynamic
you are right it kinda works for sinks though - problem is as the app scales up some TM will end up with no sink subtasks
p
problem is as the app scales up some TM will end up with no sink subtasks
Did not understand it
f
when I submit the job, there are say 10 TM, I set the max parallelism to 20 then it scales up to 15 due to load, the job is scaled up, but it's not re-submitted, so there are still only 20 subtasks for the sink
p
There is a calculation in flink for the maxParallelism / tasks. I'd say flink is smart enough to balance it out.