Hi All, what is the way to retain data in realtime table with no time limit. I configured tables with no retention time limit. But after 3,4 weeks I can see from query console that totalDocs are very few . What is the config property if we dont want to purge any data from table?

Not configuring is the correct way. If you are seeing less data, check the debug endpoint to see if there are segments that are offline for some reason. Also, if you have infinite retention, you should ensure you have enough capacity to handle infinite retention.

Debug endpoint is showing all well. Just for the knowledge, If lets say it has not enough capacity then how pinot behave, will it raise any alert or just it stops ingesting data.

Yes Pinot has metrics you can monitor and set alert on

After checking server logs , I checked this error is coming , can below exception cause message skip ? What it is about in my table config I am using flattened structure , I am using fieldsToUnnest and have multiple json nested objects in message. Although data is coming in the table from topic sometimes.

Sending request: <http://pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local:9000/segmentStoppedConsuming?reason=java.lang.RuntimeException&streamPartitionMsgOffset=149706&instance=Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098&offset=-1&name=table1__5__117__20220418T1156Z> to controller: pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown
Exception while in work
java.lang.RuntimeException: shaded.com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: org.apache.pinot.spi.data.readers.GenericRow["fieldToValueMap"]->java.util.Collections$UnmodifiableMap["$MULTIPLE_RECORDS_KEY$"]->java.util.ArrayList[0]

@User how did you solve this issue?

Bumping to 10.0.0 version have removed it but it reapear when using filterfunctions :/ filled a ticket with some info. I will provide more tomorow as asked in the ticket

Thanks.