Frantisek Hartman
03/20/2025, 8:58 AMPedro Mázala
03/20/2025, 10:15 AMthis works ok, but with growing parallelism the high number of sources and sinks create issues on our infraYou can define parallelism per operator in flink. Then you fix max parallelism to source and sink. This way you won't suffer from this and will be able to scale processing.
Frantisek Hartman
03/20/2025, 10:16 AMFrantisek Hartman
03/20/2025, 10:19 AMThe parallelism of individual operators in a job will be determined by the scheduler. It is not configurable and will be ignored if explicitly set, either on individual operators or the entire job.we are ok to abandon this, if we can get the similar behaviour with scaling up and down with number of pods
Blast Blast
03/20/2025, 10:19 AMBlast Blast
03/20/2025, 10:20 AMPedro Mázala
03/20/2025, 10:20 AMthe per operator parallelism is ignored in reactive mode of the adaptive schedulerParallelism yes. But not
maxParallelism no. I think this could be your trick.Frantisek Hartman
03/20/2025, 10:21 AMFrantisek Hartman
03/20/2025, 10:23 AMPedro Mázala
03/20/2025, 10:24 AMproblem is as the app scales up some TM will end up with no sink subtasksDid not understand it
Frantisek Hartman
03/20/2025, 10:25 AMPedro Mázala
03/20/2025, 10:27 AMPedro Mázala
03/20/2025, 10:28 AM