https://pinot.apache.org/ logo
#general
Title
# general
s

sunny

04/12/2022, 5:51 AM
Hi, I created realtime partition table.
Copy code
"tableIndexConfig": {
      "segmentPartitionConfig": {
        "columnPartitionMap": {
          "subject": {
            "functionName": "murmur",
            "numPartitions": 3
          }
        }
      },
And then add kafka topic partition (3->4) and produce data to kafka new partition. But there is no new segment in pinot. So it doesn’t show data in kafka new partition. Although changing configuration numPartitions (3->4) in pinot and rebalance servers, the result is same. It seems that there is no problem in realtime table (none partition). After adding kafka partition and then produce data to new partition, new segment is added in pinot. so It shows data in kafka new partition. Is it normal case? Otherwise, what should I check? Thanks :)
m

Mayank

04/12/2022, 4:38 PM
I am a bit unclear in the question. Are you observing that going from no partition to N partitions in an RT table there is no issue. But when you go from N partitions to N+1, the new partition doesn’t show up?
cc: @User on when does pinot pick up the new partition (is it a periodic task, or can it be triggered via rebalance)?
n

Neha Pawar

04/12/2022, 4:42 PM
periodic task, runs hourly.
@User would you please help adding all the periodic tasks and what they do to this page? https://docs.pinot.apache.org/basics/components/controller We only have all config listed here so far, which isn’t the most helpful when someone doesn’t know about the periodic tasks at all https://docs.pinot.apache.org/configuration-reference/controller#periodic-tasks-configuration
@User here’s some more info about the periodic task on the controller, which adds the new partitions: https://docs.pinot.apache.org/basics/components/controller#realtimesegmentvalidationmanager
you can also trigger manually if needed, details in the doc
m

Mark Needham

04/13/2022, 8:17 AM
n

Neha Pawar

04/13/2022, 2:29 PM
Yes.. should it have been somewhere else?
m

Mark Needham

04/13/2022, 2:30 PM
nope
just checking