https://pinot.apache.org/ logo
Channels
Powered by
k

kauts shukla

05/22/2021, 3:18 PM
m

Mayank

05/22/2021, 3:19 PM
Do you have inverted index or on-heap dictionary specified ?
In table config
k

kauts shukla

05/22/2021, 3:23 PM
@Mayank It looks like more than a GC issue, in logs everytime its lossing connection and reconnecting to zookeeper
m

Mayank

05/22/2021, 3:23 PM
That is because of GC pause, it times out to send heart beat to ZK
k

kauts shukla

05/22/2021, 3:24 PM
@Mayank Does it creates multiple connection can hit Zookeeper IOPS on call for every segemnt check.
m

Mayank

05/22/2021, 3:24 PM
No, should be single session per server
k

kauts shukla

05/22/2021, 3:24 PM
@Mayank
Copy code
"invertedIndexColumns": [
        "userid",
        "sessionid",
        "eventlabel",
        "dp_created_at",
        "timestampist"
      ]
Copy code
"sortedColumn": [
        "dp_created_at",
        "timestampist"
      ],
m

Mayank

05/22/2021, 3:24 PM
Any On-heap dictionary?
k

kauts shukla

05/22/2021, 3:25 PM
Copy code
"autoGeneratedInvertedIndex": true,
      "createInvertedIndexDuringSegmentGeneration": true, "enableDefaultStarTree": true,
      "enableDynamicStarTreeCreation": true,
m

Mayank

05/22/2021, 3:25 PM
If not, I can’t think what is occupying heap. Metadata cannot take 64GB
Oh segment generation takes heap
Are too many segments being generated in parallel?
k

kauts shukla

05/22/2021, 3:26 PM
50 segemnts in parallel
m

Mayank

05/22/2021, 3:27 PM
There you go
k

kauts shukla

05/22/2021, 3:28 PM
is this is the culprit
Copy code
"createInvertedIndexDuringSegmentGeneration": true,
m

Mayank

05/22/2021, 3:28 PM
No
Periodically segments consumed in memory are flushed to disk. This goes through some heap usage. If 50 partitions go through the same at once it will run out of heap
How did you specify 50?
k

kauts shukla

05/22/2021, 3:29 PM
kafka topic has 50 partitions
I haven’t specified its already existed
m

Mayank

05/22/2021, 3:30 PM
Not talking about consumption. Periodically the consuming segment needs to be flushed to disk, this uses some heap. Typically if all partitions flush time disk at the same time then there will be heap pressure
There is a way to specify max parallel segment generation
k

kauts shukla

05/22/2021, 3:31 PM
@Mayank: how to specify it.
m

Mayank

05/22/2021, 3:32 PM
I’ll find. In the meanwhile can you grip the log for segment generation
k

kauts shukla

05/22/2021, 3:34 PM
what i have to grep ?
m

Mayank

05/22/2021, 3:35 PM
Try something like "grep -i created segment"
k

kauts shukla

05/22/2021, 3:35 PM
post MessageLatencyMonitor it always throw ERROR [SegmentBuildTimeLeaseExtender] [pool-5-thread-1] Failed to send lease extension
m

Mayank

05/22/2021, 3:36 PM
Yeah, then it is likely segment generation
grep -i "Driver, indexing time :"
k

kauts shukla

05/22/2021, 3:37 PM
no log with this
m

Mayank

05/22/2021, 3:37 PM
In server's data dir do ls -l and see if segments have timestamp that are near by
that will tell how many generated at the same time
There should have been logs
grep -i "Trying to build segment"
or 
grep -i "Successfully built segment"
k

kauts shukla

05/22/2021, 3:41 PM
last segment created at May 22 14:08 UTC
m

Mayank

05/22/2021, 3:41 PM
how many around that time?
k

kauts shukla

05/22/2021, 3:42 PM
no luck with this grep -i “Successfully built segment”
no logs on both the server with this greo
7:38 pm IST
m

Mayank

05/22/2021, 3:43 PM
Hmm, what is uyour logging level? These are info messages and should be there for sure.
Also config 
realtime.max.parallel.segment.builds
to specify how many segment generation in parallel
k

kauts shukla

05/22/2021, 3:43 PM
logging level is INFO only
realtime.max.parallel.segment.builds ? where I should mention this
Table config ?
m

Mayank

05/22/2021, 3:44 PM
In server config
Although, I'd think that the default should not be unlimited, so still unsure if this is the root cause
Can you try setting it to a small value like 4.
If your current segment size is 1.8GB, reducing its to 112M would increase number of segments too much. May be 300MB or 500MB
Powered by