https://pinot.apache.org/ logo
#general
Title
# general
c

Cesar

10/09/2020, 10:46 PM
Hey folks, I'm doing some GC profiling of Pinot 0.40 and 0.50 and I'm noticing a huge difference between the memory allocation rate of these two versions. Everything in my setup is fixed, except the version of Pinot used. The Pinot 0.4 server process is allocating on average 4.4GB/s while Pinot 0.5 server is allocating less than 1GB/s on average. Does such huge difference make sense to you? Do you know of a code change that could have caused such a big impact? I'm using Ubuntu 20, Java 1.8 with G1GC -Xmx12G. I'm using the TPCH data set and JMeter to send 1M 'select * from tpch_lineitem' queries to Pinot.
x

Xiang Fu

10/09/2020, 10:55 PM
do you have table config?
which changed the default segment load mode from heap to mmap
c

Cesar

10/09/2020, 10:57 PM
I can get the table config. I suspected about that PR but I'm using the same segment files in both experiments. Do you still think that might be related?
s

Sidd

10/09/2020, 11:03 PM
It's not clear if the concern is higher memory allocation in general or higher memory allocation on the heap
c

Cesar

10/09/2020, 11:03 PM
@Sidd: On the heap
s

Sidd

10/09/2020, 11:03 PM
if 4.5GB is being allocated in off heap memory (direct or mmap), then it should not have any impact on GC, since that is outside JVM
Got it
c

Cesar

10/09/2020, 11:03 PM
@Xiang Fu: The table [index] config is:
Copy code
"tableIndexConfig": {
      "loadMode": "HEAP",
      "nullHandlingEnabled": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "enableDefaultStarTree": false,
      "aggregateMetrics": false,
      "autoGeneratedInvertedIndex": false
    },
s

Sidd

10/09/2020, 11:05 PM
With this config, higher heap usage is expected. The default behavior change by the PR wouldn't have impacted this table since it is hard-coded to heap
c

Cesar

10/09/2020, 11:06 PM
Yeah, agreed. What I'm puzzled is why I'm seeing such a low allocation rate in Pinot 0.50 server.
I'm trying to understand why Pinot 0.50 seem to be using so much less memory.
x

Xiang Fu

10/09/2020, 11:16 PM
do you mean 0.5.0 use less heap memory than 0.4.0 ?
c

Cesar

10/09/2020, 11:17 PM
Yeap
x

Xiang Fu

10/09/2020, 11:17 PM
hmmm
for same query, can you check if query stats are similar?
like number of docs scanned/ segments queried etc
c

Cesar

10/09/2020, 11:18 PM
Let me check and get back to you!
x

Xiang Fu

10/09/2020, 11:27 PM
sure, also do you have sample query? is it like select star with filtering ?
and order by?
c

Cesar

10/09/2020, 11:28 PM
It's just "select * from table". No ordering, no filter.
I think I found some meaningful differences on the tracing:
Pinot 0.50 "exceptions": [], "numServersQueried": 1, "numServersResponded": 1, "numSegmentsQueried": 7, "numSegmentsProcessed": 1, "numSegmentsMatched": 1, "numConsumingSegmentsQueried": 0, "numDocsScanned": 10, "numEntriesScannedInFilter": 0, "numEntriesScannedPostFilter": 160, "numGroupsLimitReached": false, "totalDocs": 49313307, "timeUsedMs": 10, "segmentStatistics": [],
Pinot 0.40 "exceptions": [], "numServersQueried": 1, "numServersResponded": 1, "numSegmentsQueried": 7, "numSegmentsProcessed": 7, "numSegmentsMatched": 7, "numConsumingSegmentsQueried": 0, "numDocsScanned": 70, "numEntriesScannedInFilter": 0, "numEntriesScannedPostFilter": 1120, "numGroupsLimitReached": false, "totalDocs": 49313307, "timeUsedMs": 225, "segmentStatistics": [],
x

Xiang Fu

10/09/2020, 11:29 PM
then I think itโ€™s this one: https://github.com/apache/incubator-pinot/pull/5686
we do early termination here
in 0.4 7 segments are processed , while in 0.5, only 1 segment is processed
c

Cesar

10/09/2020, 11:31 PM
This difference in number of segments was caused by that change? Would change anything if I add a "LIMIT/TOP" and/or some ordering to the query?
Thanks Xiang, really appreciate you helping here. I'll take a closer look at that PR. One question
x

Xiang Fu

10/09/2020, 11:33 PM
so basically the query plan will be smarter to look at query and proactively skip segment scan if there are already enough records collected
in 0.4.0, the query engine will try to collect records from all the segments then reduce
e.g. if you do
select * limit 10
, in 0.5.0 query engine will just query 1 segment and collect 10 records then return
in 0.4.0 it will collect 10 records from every segments , then reduce to 10 records
there are extra overhead on data loading and memory allocation, hence the big difference
c

Cesar

10/09/2020, 11:36 PM
Thanks a lot @Xiang Fu. That makes total sense to me now.
x

Xiang Fu

10/09/2020, 11:36 PM
๐Ÿ‘
๐Ÿ‘ 1