sunny09/22/2022, 6:30 AM
It may happen that kafka partitions are increased during operation. So Please check the issue. 😊
1. Create Pinot Partitioned Table (kafka topic partition = 3) - Query Successed. 2. Add kafka partition to a topic (3->4) - New consuming segment(3__0) is created in pinot. 3. Add kafka topic data to a new partition. - Query Successed (select * from) - But, the row is not shown in the query ( select * from where in) - The row is visible only when the segment is completed. However, the data coming into the new consuming segment(3__1) doesn't look the same as before.
Subbu Subramaniam09/22/2022, 5:26 PM
Subbu Subramaniam09/22/2022, 9:20 PM
sunny09/23/2022, 12:15 AM
sunny09/23/2022, 12:17 AM
sunny09/29/2022, 7:30 AM
However, numSegmentsQueried is equal to the total number of segments. In other words, it seems that broker pruning does not work in the Pinot partition table.
curl -u admin:verysecret -H "Content-Type: application/json" -X POST <http://pay-poc-pinot-b2.ay1.krane.9rum.cc:7001/query/sql|http://pay-poc-pinot-b2.ay1.krane.9rum.cc:7001/query/sql> -d @pinot.json | python -m json.tool
Subbu Subramaniam09/30/2022, 2:27 AM
sunny10/04/2022, 1:41 AM
Subbu Subramaniam10/04/2022, 2:43 AM
sunny10/04/2022, 6:57 AM