How does Pino's partitioning work when partitions ...
# generalc
coco
09/27/2022, 11:11 AMHow does Pino's partitioning work when partitions increase in Kafka topics? Is there any problem?
'stream ingestion with upsert'
https://docs.pinot.apache.org/basics/data-import/upsert#use-strictreplicagroup-for-routing
'routing partitioning'
https://docs.pinot.apache.org/operators/operating-pinot/tuning/routing#partitioning
n
Neha Pawar
09/27/2022, 3:57 PM
this periodic task auto detects and starts the consumption from new partitions: https://docs.pinot.apache.org/basics/components/controller#realtimesegmentvalidationmanager
Neha Pawar
09/27/2022, 3:58 PM
upsert metadata is maintained at a partition level. so change in partitioning logic would make your upserts incorrect.
c
coco
09/28/2022, 5:00 AM@Neha Pawar Yes, I know that periodic task auto detects and starts the consumption from new partitions.
My question is, if the broker gets the partition number from the condition value of the query and prunes the segment by the name of the segment, I don't think it will be able to find exactly the segments before and after the partition is added.