Hi all. I'm trying to run a Flink job for backfilling data in a k8s cluster. My job has 2 GCS filesystem sources with a lot of records each. However, it appears that Flink is only using max 2 task managers (one for each source) for reading the data from the sources, no matter how I set

parallelism

,

pipeline.max-parallelism

and

taskmanager.numberOfTaskSlots

. That leads to the fact that the job takes so long that eventually k8s terminates my pods and it has to restart from scratch. I would like to use multiple task managers to read from each source to speed it up. How can I achieve that? Why is the job manager deciding to only use one task manager per source?

Hey, what version of Flink do you use?

I'm using

1.17.1

.