Hi, Pinot creates initially the same number segmen...
# generala
Alice
04/14/2022, 5:29 AMHi, Pinot creates initially the same number segments as the number of a topic partition when ingesting a Kafka stream data to a table. Can I adjust the segment number instead of the topic partition number?
k
Kartik Khare
04/14/2022, 5:32 AMBy adjusting, you mean fix the number of segments equal to number of partitions OR just append some number suffix?
a
Alice
04/14/2022, 5:39 AMIf a topic has 100 partitions, can Pinot create just 20 segments when the table is initially created?
Alice
04/14/2022, 5:40 AMI mean, if a topic has 100 partitions, 100 partition consumers will be created, right? Can partition consumer numbers be adjusted?
k
Kartik Khare
04/14/2022, 5:40 AMNo, that wont be possible. For lowLevel consumer, each partition is consumed independently of others and goes into a different segment.
Kartik Khare
04/14/2022, 5:42 AMWe may be able to do in future with already added classes for PartitionGroups whereby few partitions can be mapped to a single consumer.
Currently, though, there is no plan for doing that
k
Kishore G
04/14/2022, 6:51 AM
Alice.. any reason why you want to do this?
Kishore G
04/14/2022, 6:52 AM
You can always merge segments into bigger segments later
Kishore G
04/14/2022, 6:52 AM
Maintaining one segment per partition has a lot of benefits
a
Alice
04/14/2022, 6:54 AMdue to limited resources, cpus. If a topic has 100 partitions, I want to create 5 tables to consume the same topic, will 500 partition consumers be created? Any suggest I can refer to design my table?
Alice
04/14/2022, 7:05 AMThanks, but how can I do this? To merge segments into bigger segments?
m
Mark Needham
04/14/2022, 9:26 AM
One way to merge them is once they're in an offline table you can use the merge rollup task - https://dev.startree.ai/docs/pinot/recipes/merge-small-segments
a
Alice
04/14/2022, 9:28 AMOk, I’ll try. Thanks. @User
a
Alice
04/14/2022, 9:49 AMThanks, that’s very helpful.
k
Kishore G
04/14/2022, 1:43 PM
See real-time to offline minion task
m
Mark Needham
04/14/2022, 1:44 PM
that one is described here - https://dev.startree.ai/docs/pinot/recipes/real-time-offline-job
2 Views