Padma Malladi
09/01/2022, 5:43 PMXiaobing
09/01/2022, 6:09 PMMayank
Padma Malladi
09/01/2022, 6:21 PMPadma Malladi
09/01/2022, 6:22 PMPadma Malladi
09/01/2022, 6:23 PMPadma Malladi
09/01/2022, 6:23 PMXiaobing
09/01/2022, 6:24 PMMayank
Mayank
CONSUMING
state, and then make a query to count the rows in that segment (you can use something like where $segmentName = '<nameOfConsumingSegment>'
Padma Malladi
09/01/2022, 6:35 PMMayank
Mayank
Padma Malladi
09/01/2022, 6:39 PMXiaobing
09/01/2022, 6:45 PMPadma Malladi
09/01/2022, 6:48 PMPadma Malladi
09/01/2022, 6:52 PMPadma Malladi
09/01/2022, 6:52 PMsegment.flush.threshold.size
Padma Malladi
09/01/2022, 6:53 PMPadma Malladi
09/01/2022, 6:53 PMrealtime.segment.flush.threshold.size
Padma Malladi
09/01/2022, 6:54 PMPadma Malladi
09/01/2022, 6:55 PMXiaobing
09/01/2022, 7:17 PMXiaobing
09/01/2022, 7:17 PMPadma Malladi
09/01/2022, 7:21 PMPadma Malladi
09/01/2022, 7:21 PMPadma Malladi
09/01/2022, 7:21 PMPadma Malladi
09/01/2022, 7:22 PMXiaobing
09/01/2022, 7:34 PMrealtime.segment.flush.threshold.size
controls when to seal a consuming segment and persist it on disk. Then a new consuming segment is created to continue to ingest data from Kafka like immediately. Basically, data is ingested continuously into Pinot.
as to segment.flush.threshold.size
I don’t know this one, where you found it?Mayank
Mayank
(where $segmaneName IN (…)
Padma Malladi
09/01/2022, 7:41 PMMayank