After adding a Kafka partition, there is a issue that data is not visible when querying the pinot partition table with a where query.

1. Create Pinot Partitioned Table (kafka topic partition = 3)
- Query Successed.

2. Add kafka partition to a topic (3->4)
- New consuming segment(3__0) is created in pinot.

3. Add kafka topic data to a new partition.
- Query Successed (select * from)
- But, the row is not shown in the query ( select * from where in)
- The row is visible only when the segment is completed. However, the data coming into the new consuming segment(3__1) doesn't look the same as before.

It may happen that kafka partitions are increased during operation. So Please check the issue. 😊

New partition should be detected automatically (either periodically or when flushing the current batch I need to check). But you may force that by applying rebalance

A better way to force it maybe to run the realtime periodic job on demand.

Perhaps a good one to document, unless already there.

ok, let me see where to add this

Thanks @Subbu Subramaniam, is that a PR on the pinot-docs repo? I didn’t find one there, if you can link it here that would help

Thank you for checking. Pinot detected new partition automatically. (new consuming segment is created) But, the problem is that the row in the new consuming segment is not shown. If I run realtime periodic job, is the row visible in where query ?

Yes, that’s what we are suggesting

Ok, If doc is updated, Please notice to me :)

The default limit is 10, you need to increase limit or run other queries to check if data is available or not

@Mayank when increasing limit, data is not visible. But, When I query(select from where) the broker, the data comes out normally. I don't know why I can't find the query when querying from the controller web.

curl -u admin:verysecret  -H "Content-Type: application/json"  -X POST   <http://pay-poc-pinot-b2.ay1.krane.9rum.cc:7001/query/sql|http://pay-poc-pinot-b2.ay1.krane.9rum.cc:7001/query/sql> -d @pinot.json  | python -m json.tool

However, numSegmentsQueried is equal to the total number of segments. In other words, it seems that broker pruning does not work in the Pinot partition table.

AFAIK if you have partitioned the ingestion topic according to a key, and then added ONE partition, things should not work, since the existing keys partition and new keys partition will not match. The only way to do this without compromising on continuous operation is to double the number of partitions in the stream (if you need more partitions). @Seunghyun and @Mayank can correct me if I am wrong.

Thanks for the reply and link :) Btw, RealtimeSegmentValidationManager is run so that pinot recognizes partitions added to Kafka. Even in that case, is there a problem with using the pinot partition table?

In a partitioned table, when pinot segments are created, the segment metadata indicates the set of keys that fall into that segment. So, when a query comes in with a certain value for the key, we look up all segments, and send the query only to those segments where the key may be a match. Now, if you suddenly change the number of partitions, then all the metadata that you have saved before for the segments will not be relevant any more, right?

IIRC, changes in number of partitions or partition function should not lead to any functional issues, the results will still be correct (we store numPartitions, as well as partition function in the segment). It is just that some optimizations for partition pruning may not apply if you changed partition function.

@Mayank @Subbu Subramaniam In summary, 1) Increasing the number of Kafka partitions has no functional issues with the pinot partition table. (result data is correct) When I tested it, if RealtimeSegmentValidationManager works, a new segment is created and the result data came out properly. However, according to Subbu, added ONE partition, things should not work, since the existing keys partition and new keys partition will not match. How can I understand what Subbu said? . 2) If the partition function is changed, optimizations such as partition pruning may not be applied. We will not change the partition function. We are only considering changing the number of partitions. Can pruning not be applied even if the number of partitions is changed? After the Kafka partition was increased and the segment was created, the query was executed for the entire segment. (numSegmentsQueried) 3) In upsert mode, the number of partitions or functions should not be changed. So do you guide me to set the number of partitions twice as large when creating upsert table? Again, this question is a bit long. Thanks for the detailed reply. I think it would be more helpful if you provide additional answers. Thanks a lot.

1. Yes, increasing partition won’t cause an issue for functionality (unless upsert). 2. Have you enabled partition pruning? If so, one segment (that was consuming at the time) will have data across multiple partitions, and due to that query routing will not be able to use partitioning, until that segment is aged out either due to retention or time boundary. 3. I recommend to have enough partitions upfront. Also note that if you had 16 partitions, the max limit for scaling servers will be 16 (one per partition) per replica, which means each server will need to hold 1/16 of the keys. You want to do the math based on this