Hi All, below attached image if our existing pipel...
# troubleshootingk
kingsathurthi
02/08/2023, 5:37 AMHi All, below attached image if our existing pipeline, where there four operator
1. Custom TCP source
2. Process
3. Map
4. sink
when ever I increase the parallelism of process,map,sink, job is running fine except the Custom TCP source. what is the approach to have parallelism in the Custom TCP source operator.
In the mean while, when I increase the parallelism for other operator new task manager is getting spawned and taskmanager is processing the data, however the custom TCP source operator is listening in only one taskmanager
d
David Anderson
02/08/2023, 3:53 PMHow is the custom source implemented? Is it a SourceFunction, a ParallelSourceFunction, or a Source?
k
kingsathurthi
02/09/2023, 9:10 AMRichParallelSourceFunction
d
David Anderson
02/10/2023, 5:17 AMHow are you adjusting the parallelism? If you modify the parallelism for the job, all of the operators should scale together, unless you’ve done something that prevents this.