While loading data from hdfs to pinot table I m getting this

Question

While loading data from hdfs to pinot table I m getting this exception r 2 apache pinot incubating 0 7 1 bin $ hadoop jar $ PINOT DISTRIBUTION DIR lib pinot all $ PINOT VERSION jar with dependencies jar org apache pinot tools admin command LaunchDataIngestionJobCommand jobSpecFile home rah executionFrameworkSpec yaml Exception in thread main java io FileNotFoundException tmp hadoop unjar7575411926296177023 shaded com google common collect ImmutableSetMultimap$EntrySet class No space left on device at java io FileOutputStream open0 Native Method at java io FileOutputStream open FileOutputStream java 270 at java io FileOutputStream lt init gt FileOutputStream java 213 at java io FileOutputStream lt init gt FileOutputStream java 162 at org apache hadoop util RunJar unJar RunJar java 110 at org apache hadoop util RunJar unJar RunJar java 85 at org apache hadoop util RunJar run RunJar java 221 at org apache hadoop util RunJar main RunJar java 148 Someone kindly suggest

Jonathan Meyer · Answer

Maybe something to do with `No space left on device`

RK · Answer

Yes < Jonathan Meyer> it s related to space issue in tmp direct pinot is creating plugin file here but I am not giving this tmp directory anywhere in my confirmation is the any way to change this tmp location I want to move this file in different location where I have enough tmp dir we don t have much space available

RK · Answer

These are the files pinot is creating at tmp location

RK · Answer

Or any way to clean this tmp location while loading data for next file I am able to load data for 3 files when I am trying for 4th day I ma getting this space issue

Jonathan Meyer · Answer

From what I read default volume size for docker containers is 10Gb there must exist a setting to increase that value Alternatively you could explicitly mount a volume which iirc would only be limited by its host storage

Jonathan Meyer · Answer

But the opinion of someone more familiar with Pinot Docker would be appreciated smile

Jonathan Meyer · Answer

if you re using docker at all actually

RK · Answer

Ok < Jonathan Meyer> I m not using docker

RK · Answer

Actually here only 943 mb space is available so I am wondering if is there any way in pinot so that we can create these tmp folder files at any other location

Laxman Ch · Answer

< RK> you can set `hadoop tmp dir` to the location you want to Default value of this property is ` tmp `

RK · Answer

Ok in which file I need to set this or its a executable directly I can run through export on terminal

Laxman Ch · Answer

You have to set this in `core site xml` Or hadoop configs can be set as a Java properties as well ` Dkey=value`

RK · Answer

Ok < Laxman Ch> here I am trying to load only 2 kB s file and space available is 943 mb But still it s giving space issue

Laxman Ch · Answer

< RK> hadoop jar command extracts all the class files into hadoop tmp directory thats how it works Some more details here

RK · Answer

Okay Thanks alot < Laxman Ch>