Hi Everyone, im facing one issue in stream data ingestion. Ive creted realtime table, where i had set segment.flush.threshold.rows to 0 and setting size to 200M But while querying on data im only able to query on totalDoc 200000, but there are more records in Kafka topic. Ive tried to set threshold size and rows, but facing same issue on another relatime tables also.

Please check server logs for any errors on why ingestion stopped.

Thankyou for reply. I've checked server logs. 1. Error logs like on segment failed to index value with Dictionary. 2. Stopping consumption due to row limit nRows=100000 But i had not set any row limit for consumption on the same table.

500M segment size under the hood is interpreted as number of rows. We start with 100k and slowly ramp up to reach 500M. HEnce you see “stopping due to row limit”. that’s expected.

Hi, Thankyou for reply, now i am able to ingest docs more then 200000, and also able to query on it. as i checked first 200000 docs were committed on path "data/controller/MassDataTable" One question, this committed data is still in-memory for query, cause as i checked my Ram usage increased gradually on both servers even records were committed on disk.

How did you fix the issue?

Hi Mayank, Checked server and controller logs as you mentioned earlier. I've found that at the time of commit Controller not able to find path, although path is provided in 'pinot-controller.conf' so i've mentioned '-dataDir <path> ' path in command as well , while restarting controller.

Thanks

Is this committed data is still in-memory for query? , cause as i checked my Ram usage increased gradually on both servers even records were committed on disk.

it will be memory mapped, and inverted indexes will be in memory.