Hi, I was performing stream ingestion through kafka in a standalone machine. I had 5 partitions created and hence 5 segments in pinot. The parameter "segment.flush.threshold.size" is set to 10000. When i try ingesting data with 100k records, only 50k records are available. Will the flushing of consuming segment take time to update or is 50k the upper bound for the configuration mentioned ?

do you mean you only see 50k records in pinot?

Size is in bytes, if you want to control rows there is s different setting

Yes, i am able to view only 50k records

hmmm. The flushing doesn't affect whether you can view them - the flushing is only when does a new segment get created. But you should be able to see the records as soon as they are ingested from kafka

What actually happens if a segment reaches max limit of any set parameter maybe rows,size or time?

the segment will be committed to the deep store and a new one will be created. The new one will be where any new messages are ingested

Okay, then why am I not able to view the entire records?

you can check which kafka offsets Pinot has processed by partition https://apache-pinot.slack.com/archives/CDRCA57FC/p1645452610220299?thread_ts=1645448815.668029&cid=CDRCA57FC

Use the table debug endpoint to check if an issues, I am wondering if the consumption stopped

the debug endpoint is here - http://localhost:9000/help#/Cluster/getTableDebugInfo You need to enter the name of the table and it will show you if there's an error

Sure, will check for it. Thank you

You can also check server log for any errors