Hmm, what's the recommended structure for calling LaunchDataIngestionJob? I'm trying to run it with 300 million rows and I'm hitting a tar size issue. I can split the DataIngestion calls but I'm curious as to what is recommended.

java.lang.RuntimeException: entry size '14879990781' is too big ( > 8589934591 )
	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumber(TarArchiveOutputStream.java:623) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76]

Logs

Typically we do hundreds MB per segment, the hard limit here is per column index size should not exceed 2gb.

Hi: Does it mean that input file can be larger than 2GB if there are N (> 1) columns as long as ANY one column index size does not exceed larger than 2GB?

Yes

Hmm, even if I try to load a small number of rows (10 million rows), I hit this issue if I have too many star tree indices.

@Jackie ^^

@Dan Hill How many columns do you have in the star-tree?

Around 7-8 dimensions and 30 metrics. All of them are current numbers. There are 18 of them.

In that case, I would recommend ~1M records per segment

Gotcha. Are there any limits to number of segments? I have roughly 40 dimensions. I can't rely on the built in time support so I have 4 date dimensions (depending on what scope is being used). I have a entity hierarchy that's about 5 levels deep. Then I have some separate star trees depending on combinations of the remaining dimensions.