https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
s

Saumya Upadhyay

04/18/2022, 3:54 AM
Hi All, what is the way to retain data in realtime table with no time limit. I configured tables with no retention time limit. But after 3,4 weeks I can see from query console that totalDocs are very few . What is the config property if we dont want to purge any data from table?
m

Mayank

04/18/2022, 3:56 AM
Not configuring is the correct way. If you are seeing less data, check the debug endpoint to see if there are segments that are offline for some reason. Also, if you have infinite retention, you should ensure you have enough capacity to handle infinite retention.
s

Saumya Upadhyay

04/18/2022, 4:52 AM
Debug endpoint is showing all well. Just for the knowledge, If lets say it has not enough capacity then how pinot behave, will it raise any alert or just it stops ingesting data.
m

Mayank

04/18/2022, 5:16 AM
Yes Pinot has metrics you can monitor and set alert on
s

Saumya Upadhyay

04/18/2022, 12:35 PM
After checking server logs , I checked this error is coming , can below exception cause message skip ? What it is about in my table config I am using flattened structure , I am using fieldsToUnnest and have multiple json nested objects in message. Although data is coming in the table from topic sometimes.
Copy code
Sending request: <http://pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local:9000/segmentStoppedConsuming?reason=java.lang.RuntimeException&streamPartitionMsgOffset=149706&instance=Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098&offset=-1&name=table1__5__117__20220418T1156Z> to controller: pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown
Exception while in work
java.lang.RuntimeException: shaded.com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: org.apache.pinot.spi.data.readers.GenericRow["fieldToValueMap"]->java.util.Collections$UnmodifiableMap["$MULTIPLE_RECORDS_KEY$"]->java.util.ArrayList[0]
m

Mayank

04/18/2022, 12:59 PM
@User how did you solve this issue?
f

francoisa

04/18/2022, 5:13 PM
Bumping to 10.0.0 version have removed it but it reapear when using filterfunctions :/ filled a ticket with some info. I will provide more tomorow as asked in the ticket
m

Mayank

04/18/2022, 5:13 PM
Thanks.