https://pinot.apache.org/ logo
r

RK

06/04/2021, 8:10 AM
While loading data from hdfs to pinot table I m getting this exception. [r-2 apache-pinot-incubating-0.7.1-bin]$ hadoop jar ${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand -jobSpecFile /home/rah/executionFrameworkSpec.yaml Exception in thread "main" java.io.FileNotFoundException: /tmp/hadoop-unjar7575411926296177023/shaded/com/google/common/collect/ImmutableSetMultimap$EntrySet.class (No space left on device)     at java.io.FileOutputStream.open0(Native Method)     at java.io.FileOutputStream.open(FileOutputStream.java:270)     at java.io.FileOutputStream.<init>(FileOutputStream.java:213)     at java.io.FileOutputStream.<init>(FileOutputStream.java:162)     at org.apache.hadoop.util.RunJar.unJar(RunJar.java:110)     at org.apache.hadoop.util.RunJar.unJar(RunJar.java:85)     at org.apache.hadoop.util.RunJar.run(RunJar.java:221)     at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Someone kindly suggest.
j

Jonathan Meyer

06/04/2021, 9:11 AM
Maybe something to do with
No space left on device
?
r

RK

06/04/2021, 9:27 AM
Yes @Jonathan Meyer it's related to space issue in /tmp direct pinot is creating plugin file here, but I am not giving this /tmp directory anywhere in my confirmation .is the any way to change this tmp location I want to move this file in different location where I have enough space.in /tmp .dir we don't have much space available
message has been deleted
These are the files pinot is creating at /tmp location
Or any way to clean this /tmp location while loading data for next file. I am able to load data for 3 files when I am trying for 4th day I ma getting this space issue
j

Jonathan Meyer

06/04/2021, 12:12 PM
From what I read, default volume size for docker containers is 10Gb, there must exist a setting to increase that value Alternatively, you could explicitly mount a volume which iirc, would only be limited by its host storage
But the opinion of someone more familiar with Pinot / Docker would be appreciated 😄
(if you're using docker at all, actually)
r

RK

06/04/2021, 1:28 PM
Ok @Jonathan Meyer I m not using docker
Actually here only 943 mb space is available so I am wondering if is there any way in pinot so that we can create these /tmp folder files at any other location.
l

Laxman Ch

06/04/2021, 1:59 PM
@RK: you can set
hadoop.tmp.dir
to the location you want to https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/core-default.xml Default value of this property is
/tmp/...
r

RK

06/04/2021, 2:01 PM
Ok in which file I need to set this or its a executable directly I can run through export on terminal.
?
l

Laxman Ch

06/04/2021, 2:08 PM
You have to set this in
core-site.xml
Or hadoop configs can be set as a Java properties as well
-Dkey=value
👍 1
r

RK

06/04/2021, 2:37 PM
Ok @Laxman Ch here I am trying to load only 2 kB's file and space available is 943 mb . But still it's giving space issue ?
l

Laxman Ch

06/05/2021, 6:53 PM
@RK: hadoop jar command extracts all the class files into hadoop tmp directory. thats how it works. Some more details here http://mail-archives.apache.org/mod_mbox/hadoop-user/201410.mbox/%3CCAOcnVr2bd5mq+tEyrzpY5WWsaiLuvikme+qNVxUmiA4RvfTMkA@mail.gmail.com%3E
r

RK

06/06/2021, 4:33 PM
Okay Thanks alot @Laxman Ch