I m trying to track down where my SegmentCreation job is spe

Question

I m trying to track down where my SegmentCreation job is spending time It s been running for a few hours but here are the only time measurements I see ```Assembled and processed 7990100 records from 17 columns in 121766 ms 65 618484 rec ms 1115 5142 cell ms time spent so far 0 reading 237 ms and 99 processing 121766 ms at row 7990100 reading next block block read in memory in 44 ms row count = 1418384 Finished building StatsCollector Collected stats for 9408484 documents Created dictionary for INT column RecordReader initialized will read a total of 9408484 records at row 0 reading next block Got brand new decompressor gz block read in memory in 133 ms row count = 7990100 Start building IndexCreator Assembled and processed 7990100 records from 17 columns in 127060 ms 62 884464 rec ms 1069 0359 cell ms time spent so far 0 reading 133 ms and 99 processing 127060 ms at row 7990100 reading next block block read in memory in 26 ms row count = 1418384 Finished records indexing in IndexCreator Finished segment seal Generated 25884 star tree records from 9408484 segment records Finished constructing star tree got 1228 tree nodes and 2058 records under star node Finished creating aggregated documents got 1227 aggregated records Finished building star tree in 276631ms Starting building star tree with config StarTreeV2BuilderConfig Generated 9408484 star tree records from 9408484 segment records``` It s been stuck at that last line for a long time but still seems to be doing something All the time measurements only add up to a few minutes Any idea where it s spending time

Ken Krugler · Answer

There was a similar question recently in < CDRCA57FC|general> where GCing could be a potential source of slowness trying to include a ref let s see if this works

Xiang Fu · Answer

I feel it s mostly spent on startree generation

Aaron Wishnick · Answer

Awesome thanks for the pointer I am running with the default heap size so that could be a culprit

Aaron Wishnick · Answer

How can I override the default JVM heap settings if I m using the docker image Is the right fix for me to override those heap settings or is that the wrong fix here

Aaron Wishnick · Answer

I see pinot running the data ingestion job with ` Xms4G`

Ken Krugler · Answer

Hi < Aaron Wishnick> sorry don t know best approach with Docker I run the ingestion job on a bare metal server so just changed what was in the script used to launch that

Ken Krugler · Answer

In `bin pinot admin sh` I changed one line to ``` ALL JAVA OPTS= Xms15G Xmx15G Dlog4j2 configurationFile=conf pinot admin log4j2 xml ```

Aaron Wishnick · Answer

Thanks

Aaron Wishnick · Answer

I m no Java expert but this looks like really bad GC overhead right ```2021 05 11T20 38 54 205+0000 Full GC Ergonomics PSYoungGen 3348480K gt 2829207K 4470272K ParOldGen 11184779K gt 11184779K 11185152K 14533259K gt 14013987K 15655424K Metaspace 55133K gt 55133K 1099776K secs Times user=31738 44 sys=70 66 real=546 08 secs 2021 05 11T20 48 00 653+0000 Full GC Ergonomics PSYoungGen 3348480K gt 2829208K 4470272K ParOldGen 11184779K gt 11184779K 11185152K 14533259K gt 14013987K 15655424K Metaspace 55133K gt 55133K 1099776K secs Times user=28270 29 sys=112 95 real=522 62 secs 2021 05 11T20 56 43 693+0000 Full GC Ergonomics PSYoungGen 3348480K gt 2829208K 4470272K ParOldGen 11184779K gt 11184779K 11185152K 14533259K gt 14013987K 15655424K Metaspace 55133K gt 55133K 1099776K secs Times user=35688 72 sys=23 60 real=563 44 secs ```

Ken Krugler · Answer

Yes that s pretty bad

Aaron Wishnick · Answer

Is the fix to make the heap bigger I m using a 16 G heap right now Or are there other data ingest settings I can tune to reduce memory usage

Aaron Wishnick · Answer

If it s relevant the segment I m ingesting is comprised of ~80 different parquet files any chance that pinot is doing something funky with parallelism like spinning up a job per parquet file in parallel so I ve got 80 jobs I have the default settings there

Ken Krugler · Answer

The job file should have a parallelism setting in it `segmentCreationJobParallelism`

Ken Krugler · Answer

The recommendation is to set this equal to the number of cores but if you re processing large input files you might need to reduce the parallelism

Ken Krugler · Answer

Generating star tree indexes can use significant amounts of memory so when I added some of these to existing tables I had to do exactly that from 16 down to 4

Aaron Wishnick · Answer

Ok thanks I ll give that a shot

Aaron Wishnick · Answer

The machine I m running on has 40 cores and it does look like every core is maxed out right now

Ken Krugler · Answer

That would do it slightly smiling face

Aaron Wishnick · Answer

grimacing

Ken Krugler · Answer

I should update the docs to cover the above issues

Aaron Wishnick · Answer

Do you have any guidance around heap sizing I do have plenty more memory but I m aware that sometimes making the heap bigger makes things worse

Ken Krugler · Answer

I would leave the heap at 16gb and reduce parallelism to say 4 just as a test

Aaron Wishnick · Answer

That d be helpful I noticed that the only place segmentCreationJobParallelism is mentioned is <https docs pinot apache org users tutorials use s3 and pinot in docker> the pages on job specs don t mention it

Aaron Wishnick · Answer

Got it let s see how that goes

Aaron Wishnick · Answer

Heh as far as I can tell this is going way faster so far

Aaron Wishnick · Answer

I ll confirm once these jobs finish

Ken Krugler · Answer

600 second full GCs tend to slow things down slightly smiling face

Aaron Wishnick · Answer

slightly smiling face

Aaron Wishnick · Answer

Ok this is better but not great ```2021 05 12T16 25 48 464+0000 Full GC Ergonomics PSYoungGen 499972K gt 0K 4527104K ParOldGen 10669128K gt 11093033K 11185152K 11169101K gt 11093033K 15712256K Metaspace 55592K gt 55592K 1099776K 36 2899382 secs Times user=962 56 sys=0 69 real=36 29 secs 2021 05 12T16 26 28 268+0000 Full GC Ergonomics PSYoungGen 3428864K gt 228298K 4527104K ParOldGen 11093033K gt 11170454K 11185152K 14521897K gt 11398752K 15712256K Metaspace 55592K gt 55592K 1099776K 51 5969673 secs Times user=1290 02 sys=9 49 real=51 60 secs 2021 05 12T16 27 27 406+0000 Full GC Ergonomics PSYoungGen 3428864K gt 496053K 4527104K ParOldGen 11170454K gt 11170454K 11185152K 14599318K gt 11666507K 15712256K Metaspace 55592K gt 55592K 1099776K 43 8830724 secs Times user=936 58 sys=13 11 real=43 88 secs ```

Aaron Wishnick · Answer

Is the next move to increase my heap size Or is there anything else I should do Am I trying to process too much data and I should break my parquet files up into smaller pieces or something Or other JVM GC settings that help with this

Ken Krugler · Answer

How big is a typical segment you re generating You could take one of these unpack it and example the metadata file to see how big the various columns are That would give you some indication of whether you should generate smaller input files

Aaron Wishnick · Answer

The input is about 80 parquet files 16 GB in total

Ken Krugler · Answer

That sounds smaller than what I d expect But Parquet can compress data down a lot

Xiang Fu · Answer

I somehow feel the segment itself is ok startree itself may explode the memory ```Generated 25884 star tree records from 9408484 segment records Finished constructing star tree got 1228 tree nodes and 2058 records under star node Finished creating aggregated documents got 1227 aggregated records Finished building star tree in 276631ms Starting building star tree with config StarTreeV2BuilderConfig Generated 9408484 star tree records from 9408484 segment records```

Xiang Fu · Answer

do you have startree configs and one segment metadata

Ken Krugler · Answer

Yes `Generated 9408484 star tree records from 9408484 segment records` seems odd maybe trying to use a column in the star tree definition which is unique per row If so that won t help your queries

Aaron Wishnick · Answer

Nice catch

Aaron Wishnick · Answer

So it looks like one issue was I had `enableDefaultStarTree` set to true so it was generating a second star tree across all dimensions and yes one of those dimensions could have been unique

Aaron Wishnick · Answer

I now see `Generated 25884 star tree records from 9408484 segment records`