While loading data from hdfs to pinot table I m ge...
# troubleshootingr
RK
06/04/2021, 8:10 AMWhile loading data from hdfs to pinot table I m getting this exception.
[r-2 apache-pinot-incubating-0.7.1-bin]$ hadoop jar ${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand -jobSpecFile /home/rah/executionFrameworkSpec.yaml
Exception in thread "main" java.io.FileNotFoundException: /tmp/hadoop-unjar7575411926296177023/shaded/com/google/common/collect/ImmutableSetMultimap$EntrySet.class (No space left on device)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.hadoop.util.RunJar.unJar(RunJar.java:110)
at org.apache.hadoop.util.RunJar.unJar(RunJar.java:85)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Someone kindly suggest.
j
Jonathan Meyer
06/04/2021, 9:11 AMMaybe something to do with
No space left on devicer
RK
06/04/2021, 9:27 AMYes @Jonathan Meyer it's related to space issue in /tmp direct pinot is creating plugin file here, but I am not giving this /tmp directory anywhere in my confirmation .is the any way to change this tmp location I want to move this file in different location where I have enough space.in /tmp .dir we don't have much space available
RK
06/04/2021, 9:28 AMIMG_20210604_113822.jpg
RK
06/04/2021, 9:28 AMThese are the files pinot is creating at /tmp location
RK
06/04/2021, 9:29 AMOr any way to clean this /tmp location while loading data for next file. I am able to load data for 3 files when I am trying for 4th day I ma getting this space issue
j
Jonathan Meyer
06/04/2021, 12:12 PMFrom what I read, default volume size for docker containers is 10Gb, there must exist a setting to increase that value
Alternatively, you could explicitly mount a volume which iirc, would only be limited by its host storage
Jonathan Meyer
06/04/2021, 12:13 PMBut the opinion of someone more familiar with Pinot / Docker would be appreciated 😄
Jonathan Meyer
06/04/2021, 12:13 PM(if you're using docker at all, actually)
r
RK
06/04/2021, 1:28 PMOk @Jonathan Meyer I m not using docker
RK
06/04/2021, 1:32 PMActually here only 943 mb space is available so I am wondering if is there any way in pinot so that we can create these /tmp folder files at any other location.
l
Laxman Ch
06/04/2021, 1:59 PM@RK: you can set
hadoop.tmp.dir/tmp/...r
RK
06/04/2021, 2:01 PMOk in which file I need to set this or its a executable directly I can run through export on terminal.
RK
06/04/2021, 2:01 PM?
l
Laxman Ch
06/04/2021, 2:08 PMYou have to set this in
core-site.xml-Dkey=value👍 1
r
RK
06/04/2021, 2:37 PMOk @Laxman Ch here I am trying to load only 2 kB's file and space available is 943 mb . But still it's giving space issue ?
l
Laxman Ch
06/05/2021, 6:53 PM@RK: hadoop jar command extracts all the class files into hadoop tmp directory. thats how it works. Some more details here
http://mail-archives.apache.org/mod_mbox/hadoop-user/201410.mbox/%3CCAOcnVr2bd5mq+tEyrzpY5WWsaiLuvikme+qNVxUmiA4RvfTMkA@mail.gmail.com%3E
r
RK
06/06/2021, 4:33 PMOkay Thanks alot @Laxman Ch