https://pinot.apache.org/ logo
Join Slack
Powered by
Hi everyone , recently I tried to upgrade pinot ve...
# troubleshooting
a
Hi everyone , recently I tried to upgrade pinot version from 0.7.1 and I am doing ingestion using files stored in azure blob but I am getting this error
Copy code
java.lang.IllegalStateException: Unable to extract out the relative path for input file file path "file path"
in 0.8.0 and 0.9.0 but it is working fine in 0.7.1
m
Can you share the stack trace and your ingestion job spec file?
a
this is general blue print of ingestion.yaml
Copy code
executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '<adl://path/to/input/directory/>'
outputDirURI: '<adl://path/to/output/directory/>'
overwriteOutput: true
pinotFSSpecs:
    - scheme: adl
      className: org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS
      configs:
        accountName: ''
        accessKey: ''
        fileSystemName: ''
recordReaderSpec:
    dataFormat: 'avro'
    className: 'org.apache.pinot.plugin.inputformat.orc.ORCRecordReader'
tableSpec:
    tableName: 'TABLE_NAME'
pinotClusterSpecs:
    - controllerURI: 'controller_uri'
and stacktrace is
Copy code
Failed to generate Pinot segment for file - file:/TABLE_NAME/FILE1
java.lang.IllegalStateException: Unable to extract out the relative path for input file 'file:/TABLE_NAME/FILE1', based on base i
nput path: <adl://path/to/TABLE_NAME/>
        at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b8
8469a8dbae742a4b609808]
        at org.apache.pinot.common.segment.generation.SegmentGenerationUtils.getRelativeOutputPath(SegmentGenerationUtils.java:156) ~[pinot-all-0.8.0-jar-with
-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(SegmentGenerationJobRunner.java:276) ~[
pinot-batch-ingestion-standalone-0.8.0-shaded.jar:0.8.0-9a0f41bc24243ff74315723b0153b534c2596e30]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
p.s same ingestion.yaml is working fine in 0.7.1
m
What do you see in the log for this message:
Copy code
<http://LOGGER.info|LOGGER.info>("Initializing PinotFS for scheme {}, classname {}", scheme, fsClassName);
Also, your spec says 
dataFormat: 'avro'
, but you are using 
'org.apache.pinot.plugin.inputformat.orc.ORCRecordReader'
?
a
No luck changing dataFormat: 'avro' to dataFormat: 'orc'
Copy code
Initializing PinotFS for scheme adl, classname org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS                                                             
Authenticating using the access key to the account.                                                                                                           
ADLSGen2PinotFS is initialized (accountName=accountName, fileSystemName=fileSystemName, dfsServiceEndpointUrl=dfsServiceEndpointUrl, blobServiceEndpointUrl=blobServiceEndpointUrl, enableChecksum=false)
m
@Seunghyun ^^
s
@Mayank It looks that the exception is thrown during relative path computation. I can easily reproduce the issue
Copy code
public static void main(String[] args) {
    getRelativeOutputPath(URI.create("<adl://path/to/TABLE_NAME>"), URI.create("file:/TABLE_NAME/FILE1"), URI.create("<adl://path/to/OUTPUT_TABLE_NAME>"));
  }
x
right, the scheme should be 
adl
not 
file
s
Somehow the 
inputFile
is passed as 
file:/TABLE_NAME/FILE1
not 
<adl://path/to/TABLE_NAME/FILE1>
x
hmm, that’s weird, it means the inputFileURI scheme is not adl but file
m
Yes I suspected that too, that’s why I asked for log but is shows above adls gen2 was initialized
x
it means this 
String[] files = _inputDirFS.listFiles(_inputDirURI, true);
is giving all the uris with 
file
scheme?
is it because adl PinotFs listFiles doesn’t give scheme?
s
oh yeah probably we don’t put the prefix
m
Is this a new issue then?
x
I think once we saw this for hdfs but not checked adls
Open in Slack
PreviousNext
Powered by