Hi I created realtime partition table ``` tableIndexConfig s Apache Pinot #general

Hi, I created realtime partition table. ```"tableI...

sunny

04/12/2022, 5:51 AM

Hi, I created realtime partition table.

Copy code

"tableIndexConfig": {
      "segmentPartitionConfig": {
        "columnPartitionMap": {
          "subject": {
            "functionName": "murmur",
            "numPartitions": 3
          }
        }
      },

And then add kafka topic partition (3->4) and produce data to kafka new partition. But there is no new segment in pinot. So it doesn’t show data in kafka new partition. Although changing configuration numPartitions (3->4) in pinot and rebalance servers, the result is same. It seems that there is no problem in realtime table (none partition). After adding kafka partition and then produce data to new partition, new segment is added in pinot. so It shows data in kafka new partition. Is it normal case? Otherwise, what should I check? Thanks :)

Mayank

04/12/2022, 4:38 PM

I am a bit unclear in the question. Are you observing that going from no partition to N partitions in an RT table there is no issue. But when you go from N partitions to N+1, the new partition doesn’t show up?

Mayank

04/12/2022, 4:39 PM

cc: @User on when does pinot pick up the new partition (is it a periodic task, or can it be triggered via rebalance)?

Neha Pawar

04/12/2022, 4:42 PM

periodic task, runs hourly.

Neha Pawar

04/12/2022, 4:44 PM

@User would you please help adding all the periodic tasks and what they do to this page? https://docs.pinot.apache.org/basics/components/controller We only have all config listed here so far, which isn’t the most helpful when someone doesn’t know about the periodic tasks at all https://docs.pinot.apache.org/configuration-reference/controller#periodic-tasks-configuration

Neha Pawar

04/13/2022, 12:23 AM

@User here’s some more info about the periodic task on the controller, which adds the new partitions: https://docs.pinot.apache.org/basics/components/controller#realtimesegmentvalidationmanager

Neha Pawar

04/13/2022, 12:24 AM

you can also trigger manually if needed, details in the doc

Mark Needham

04/13/2022, 8:17 AM

@User I think you added them to this page instead? https://docs.pinot.apache.org/basics/components/controller#controller-periodic-tasks

Neha Pawar

04/13/2022, 2:29 PM

Yes.. should it have been somewhere else?