Hi everyone, can someone confirm my understanding ...
# troubleshootinga
Artun Duman
02/10/2023, 6:07 PMHi everyone, can someone confirm my understanding of
SplitEnumeratorm
Martijn Visser
02/10/2023, 6:09 PMWhy wouldn’t it be possible to increase the number of partitions? Have you checked https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.html and https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/connector/source/SplitEnumerator.html
Martijn Visser
02/10/2023, 6:10 PMReading is done by TaskManagers, not by the JobManager
k
Krzysztof Chmielewski
02/10/2023, 6:23 PMGenerally speaking (not for Kafka Source), the TaskManager could be a bottleneck if Split discovery and split creation logic would be an overwhelming task.
Krzysztof Chmielewski
02/10/2023, 6:27 PMIf Im correct, for Kafka Source Split = Kafka topic, so for Kafka Source Split is just a "descripor" of a topic, that is send to readers. Kafka Source can discover new partitions (craete new Splits) I dont think this could cause significatn delays in this case
a
Artun Duman
02/10/2023, 6:30 PMOh apologies, I think I had a conceptual misunderstanding when I checked the diagram here: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/#data-source-concepts
And I thought the round-robin here meant that the actual data was being load balanced. Then I think it means that one Kafka partition is a split in Flink and which task manager reads what split is decided by the JobManager, is that correct?
m
Martijn Visser
02/10/2023, 6:30 PMA split at Kafka is actually a partition IIRC
Martijn Visser
02/10/2023, 6:31 PMI can see how that can be confusing yes
a
Artun Duman
02/10/2023, 6:33 PMI think I didn't pay close attention to the docs, apologies 😅 Does my current understanding make sense? Therefore, is it safe to assume that we can't have more TaskManagers reading from a topic than the number of partitions in that topic?
m
Martijn Visser
02/10/2023, 6:33 PMThis explains the behind the scenes: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka/#behind-the-scene
Martijn Visser
02/10/2023, 6:35 PMYou can have more TaskManagers: they won’t be reading from a Kafka partition in that case
a
Artun Duman
02/10/2023, 6:35 PMRight, thank you for the explanation! 🙏
t
Take Care
03/01/2023, 4:10 AM@Martijn Visser I have a quick follow up regarding your last comment, Is it possible to extend SplitFetch to use 2 task managers to poll 1 kafka partition concurrently with different offset range?
m
Martijn Visser
03/02/2023, 2:42 PM@Take Care I don't think so, because I could see that cause correctness issues when you need to restore from a snapshot
5 Views