I need your help with flink custom source. I have ...
# troubleshooting
u
I need your help with flink custom source. I have a custom BOUNDED source, i want to specify to him when to stop reading. The current way I did it is by self implementing the pollNext function of SourceReaderBase, but it works only for parallelsim of 1... There is good practice way what to override in order to specify a custom bounded source when to stop reading? Thanks a lot!
p
Hey there! How are you doing that on the
pollNext
? You'd also have the possibility to use
isEndOfStream
on your deserializer. How are you willing to control the stop of the stream? Specific record? And why it only works out with
parallelism=1
? Is it only killing the task that ``pollNext`` stops it?
Do you mind sharing the code you use on
pollNext
? Perhaps you need to adapt it to
amount of splits / parallelism
due to each task taking over part of the process. This would be the way only if the data is evenly spread across each parallel task tho