https://pinot.apache.org/ logo
Join Slack
Powered by
Hey folks, I'm doing some GC profiling of Pinot 0....
# general
c
Hey folks, I'm doing some GC profiling of Pinot 0.40 and 0.50 and I'm noticing a huge difference between the memory allocation rate of these two versions. Everything in my setup is fixed, except the version of Pinot used. The Pinot 0.4 server process is allocating on average 4.4GB/s while Pinot 0.5 server is allocating less than 1GB/s on average. Does such huge difference make sense to you? Do you know of a code change that could have caused such a big impact? I'm using Ubuntu 20, Java 1.8 with G1GC -Xmx12G. I'm using the TPCH data set and JMeter to send 1M 'select * from tpch_lineitem' queries to Pinot.
x
do you have table config?
which changed the default segment load mode from heap to mmap
c
I can get the table config. I suspected about that PR but I'm using the same segment files in both experiments. Do you still think that might be related?
s
It's not clear if the concern is higher memory allocation in general or higher memory allocation on the heap
c
@Sidd: On the heap
s
if 4.5GB is being allocated in off heap memory (direct or mmap), then it should not have any impact on GC, since that is outside JVM
Got it
c
@Xiang Fu: The table [index] config is:
Copy code
"tableIndexConfig": {
      "loadMode": "HEAP",
      "nullHandlingEnabled": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "enableDefaultStarTree": false,
      "aggregateMetrics": false,
      "autoGeneratedInvertedIndex": false
    },
s
With this config, higher heap usage is expected. The default behavior change by the PR wouldn't have impacted this table since it is hard-coded to heap
c
Yeah, agreed. What I'm puzzled is why I'm seeing such a low allocation rate in Pinot 0.50 server.
I'm trying to understand why Pinot 0.50 seem to be using so much less memory.
x
do you mean 0.5.0 use less heap memory than 0.4.0 ?
c
Yeap
x
hmmm
for same query, can you check if query stats are similar?
like number of docs scanned/ segments queried etc
c
Let me check and get back to you!
x
sure, also do you have sample query? is it like select star with filtering ?
and order by?
c
It's just "select * from table". No ordering, no filter.
I think I found some meaningful differences on the tracing:
Pinot 0.50 "exceptions": [], "numServersQueried": 1, "numServersResponded": 1, "numSegmentsQueried": 7, "numSegmentsProcessed": 1, "numSegmentsMatched": 1, "numConsumingSegmentsQueried": 0, "numDocsScanned": 10, "numEntriesScannedInFilter": 0, "numEntriesScannedPostFilter": 160, "numGroupsLimitReached": false, "totalDocs": 49313307, "timeUsedMs": 10, "segmentStatistics": [],
Pinot 0.40 "exceptions": [], "numServersQueried": 1, "numServersResponded": 1, "numSegmentsQueried": 7, "numSegmentsProcessed": 7, "numSegmentsMatched": 7, "numConsumingSegmentsQueried": 0, "numDocsScanned": 70, "numEntriesScannedInFilter": 0, "numEntriesScannedPostFilter": 1120, "numGroupsLimitReached": false, "totalDocs": 49313307, "timeUsedMs": 225, "segmentStatistics": [],
x
we do early termination here
in 0.4 7 segments are processed , while in 0.5, only 1 segment is processed
c
This difference in number of segments was caused by that change? Would change anything if I add a "LIMIT/TOP" and/or some ordering to the query?
Thanks Xiang, really appreciate you helping here. I'll take a closer look at that PR. One question
x
so basically the query plan will be smarter to look at query and proactively skip segment scan if there are already enough records collected
in 0.4.0, the query engine will try to collect records from all the segments then reduce
e.g. if you do 
select * limit 10
, in 0.5.0 query engine will just query 1 segment and collect 10 records then return
in 0.4.0 it will collect 10 records from every segments , then reduce to 10 records
there are extra overhead on data loading and memory allocation, hence the big difference
c
Thanks a lot @Xiang Fu. That makes total sense to me now.
x
👍
👍 1
Open in Slack
PreviousNext
Powered by