https://pinot.apache.org/ logo
Join Slack
Powered by
<https://apache-pinot.slack.com/archives/CDRCA57FC...
# troubleshooting
k
m
Do you have inverted index or on-heap dictionary specified ?
In table config
k
@Mayank It looks like more than a GC issue, in logs everytime its lossing connection and reconnecting to zookeeper
m
That is because of GC pause, it times out to send heart beat to ZK
k
@Mayank Does it creates multiple connection can hit Zookeeper IOPS on call for every segemnt check.
m
No, should be single session per server
k
@Mayank
Copy code
"invertedIndexColumns": [
        "userid",
        "sessionid",
        "eventlabel",
        "dp_created_at",
        "timestampist"
      ]
Copy code
"sortedColumn": [
        "dp_created_at",
        "timestampist"
      ],
m
Any On-heap dictionary?
k
Copy code
"autoGeneratedInvertedIndex": true,
      "createInvertedIndexDuringSegmentGeneration": true, "enableDefaultStarTree": true,
      "enableDynamicStarTreeCreation": true,
m
If not, I can’t think what is occupying heap. Metadata cannot take 64GB
Oh segment generation takes heap
Are too many segments being generated in parallel?
k
50 segemnts in parallel
m
There you go
k
is this is the culprit
Copy code
"createInvertedIndexDuringSegmentGeneration": true,
m
No
Periodically segments consumed in memory are flushed to disk. This goes through some heap usage. If 50 partitions go through the same at once it will run out of heap
How did you specify 50?
k
kafka topic has 50 partitions
I haven’t specified its already existed
m
Not talking about consumption. Periodically the consuming segment needs to be flushed to disk, this uses some heap. Typically if all partitions flush time disk at the same time then there will be heap pressure
There is a way to specify max parallel segment generation
k
@Mayank: how to specify it.
m
I’ll find. In the meanwhile can you grip the log for segment generation
k
what i have to grep ?
m
Try something like "grep -i created segment"
k
post MessageLatencyMonitor it always throw ERROR [SegmentBuildTimeLeaseExtender] [pool-5-thread-1] Failed to send lease extension
m
Yeah, then it is likely segment generation
grep -i "Driver, indexing time :"
k
no log with this
m
In server's data dir do ls -l and see if segments have timestamp that are near by
that will tell how many generated at the same time
There should have been logs
grep -i "Trying to build segment"
or 
grep -i "Successfully built segment"
k
last segment created at May 22 14:08 UTC
m
how many around that time?
k
no luck with this grep -i “Successfully built segment”
no logs on both the server with this greo
7:38 pm IST
m
Hmm, what is uyour logging level? These are info messages and should be there for sure.
Also config 
realtime.max.parallel.segment.builds
to specify how many segment generation in parallel
k
logging level is INFO only
realtime.max.parallel.segment.builds ? where I should mention this
Table config ?
m
In server config
Although, I'd think that the default should not be unlimited, so still unsure if this is the root cause
Can you try setting it to a small value like 4.
If your current segment size is 1.8GB, reducing its to 112M would increase number of segments too much. May be 300MB or 500MB
Open in Slack
PreviousNext
Powered by