Hi I got this issue when submit spark job to ingest batch fi

Question

Hi I got this issue when submit spark job to ingest batch file ```21 04 20 03 03 42 ERROR org apache pinot tools admin command LaunchDataIngestionJobCommand Got exception to kick off standalone data ingestion job java lang RuntimeException Caught exception during running org apache pinot plugin ingestion batch spark SparkSegmentGenerationJobRunner at org apache pinot spi ingestion batch IngestionJobLauncher kickoffIngestionJob IngestionJobLauncher java 144 at org apache pinot spi ingestion batch IngestionJobLauncher runIngestionJob IngestionJobLauncher java 117 at org apache pinot tools admin command LaunchDataIngestionJobCommand execute LaunchDataIngestionJobCommand java 132 at org apache pinot tools admin command LaunchDataIngestionJobCommand main LaunchDataIngestionJobCommand java 67 at sun reflect NativeMethodAccessorImpl invoke0 Native Method at sun reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62 at sun reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43 at java lang reflect Method invoke Method java 498 at org apache spark deploy JavaMainApplication start SparkApplication scala 52 at <http org apache spark deploy SparkSubmit org|org apache spark deploy SparkSubmit org>$apache$spark$deploy$SparkSubmit$$runMain SparkSubmit scala 951 at org apache spark deploy SparkSubmit doRunMain$1 SparkSubmit scala 180 at org apache spark deploy SparkSubmit submit SparkSubmit scala 203 at org apache spark deploy SparkSubmit doSubmit SparkSubmit scala 90 at org apache spark deploy SparkSubmit$$anon$2 doSubmit SparkSubmit scala 1030 at org apache spark deploy SparkSubmit$ main SparkSubmit scala 1039 at org apache spark deploy SparkSubmit main SparkSubmit scala Caused by java nio file FileSystemNotFoundException Provider gs not installed at java nio file Paths get Paths java 147 at org apache pinot plugin filesystem GcsPinotFS copy GcsPinotFS java 262 at org apache pinot plugin ingestion batch spark SparkSegmentGenerationJobRunner run SparkSegmentGenerationJobRunner java 344 at org apache pinot spi ingestion batch IngestionJobLauncher kickoffIngestionJob IngestionJobLauncher java 142 15 more```

Phúc Huỳnh · Answer

after some deep dive I think it s the Scala filesystem only 2 providers file jar

Phúc Huỳnh · Answer

base on some explain amp discuss <https stackoverflow com questions 39500445 filesystem provider disappearing in spark> I suggest move `java nio file Path` to `org apache hadoop fs Path` Have any idea

Jackie · Answer

< Xiang Fu> ^^

Xiang Fu · Answer

can you try to add ```google cloud nio``` dependency into pinot gcs pom file

Xiang Fu · Answer

Xiang Fu · Answer

hmm seems there is already one in pom ``` lt dependency gt lt groupId gt com google cloud lt groupId gt lt artifactId gt google cloud nio lt artifactId gt lt version gt 0 120 0 alpha lt version gt lt dependency gt ```

Phúc Huỳnh · Answer

yup in pom already has dependency

Xiang Fu · Answer

I m looking at here <https github com googleapis google cloud java blob v0 120 0 google cloud clients google cloud contrib google cloud nio src main java com google cloud storage contrib nio package info java|https github com googleapis google cloud java blob v0 120 0 google cloud clients ain java com google cloud storage contrib nio package info java>

Xiang Fu · Answer

maybe we can modify this code to directly use google cloudstorage api

Xiang Fu · Answer

does this code work on your standalone application

Xiang Fu · Answer

or test

Xiang Fu · Answer

but only failed on spark

Phúc Huỳnh · Answer

yup

Phúc Huỳnh · Answer

this code work on standalone application

Phúc Huỳnh · Answer

but fail on spark cluster on dataproc

Xiang Fu · Answer

hmm

Xiang Fu · Answer

on spark cluster can you try to put the gcs plugin to the classpath

Phúc Huỳnh · Answer

already tried smile

Xiang Fu · Answer

I assume this is java 8

Xiang Fu · Answer

or 11

Phúc Huỳnh · Answer

java 8

Phúc Huỳnh · Answer

let me check image spark <https cloud google com dataproc docs concepts versioning dataproc release 2 0>

Xiang Fu · Answer

another way is to shade pinot ingestion spark and pinot gcs together into one jar

Phúc Huỳnh · Answer

smile another way to fix remove `java nio file Paths` in gcs pinot then custom another Paths get function

Xiang Fu · Answer

right

Xiang Fu · Answer

then you need to implement the relatize function there

Xiang Fu · Answer

I feel better to just use native google gcs lib

Phúc Huỳnh · Answer

well just another error ini sendSegmentUri pharse ```Caused by java lang IllegalStateException PinotFS for scheme gs has not been initialized at shaded com google common base Preconditions checkState Preconditions java 518 at org apache pinot spi filesystem PinotFSFactory create PinotFSFactory java 80 at org apache pinot plugin ingestion batch common SegmentPushUtils sendSegmentUris SegmentPushUtils java 158 at org apache pinot plugin ingestion batch spark SparkSegmentUriPushJobRunner$1 call SparkSegmentUriPushJobRunner java 122 at org apache pinot plugin ingestion batch spark SparkSegmentUriPushJobRunner$1 call SparkSegmentUriPushJobRunner java 117 ```

Phúc Huỳnh · Answer

are you have any idea how to fix it

Xiang Fu · Answer

It s missing fs init in the code <https github com apache incubator pinot pull 6819>

Xiang Fu · Answer

merged the above pr can you try it again

Phúc Huỳnh · Answer

hmm i try it again but still errors

Phúc Huỳnh · Answer

i guess i already find issue i forget add dir plugins property in spark job But i m found another issue when packing plugins to workers node

Phúc Huỳnh · Answer

```21 04 22 07 25 48 ERROR org apache pinot plugin ingestion batch spark SparkSegmentGenerationJobRunner Failed to tar plugins directory java io IOException Request to write 4096 bytes exceeds size in header of 12453302 bytes for entry pinot plugins tar gz at org apache commons compress archivers tar TarArchiveOutputStream write TarArchiveOutputStream java 449 ```

Xiang Fu · Answer

hmm seems like an issue of tar gz everything in plugin dir to a file

Xiang Fu · Answer

I will take a look

Xiang Fu · Answer

are you using default plugin dir or you ve put more files into that

Xiang Fu · Answer

do you set this `plugins dir` in your java cmd

Xiang Fu · Answer

I was testing this util locally ``` public static void main String args try TarGzCompressionUtils createTarGzFile new File Users xiangfu workspace pinot dev pinot distribution target apache pinot incubating 0 8 0 SNAPSHOT bin apache pinot incubating 0 8 0 SNAPSHOT bin plugins new File tmp plugin tar gz catch IOException e e printStackTrace ```

Xiang Fu · Answer

it works and cannot reproduce the issue

Phúc Huỳnh · Answer

i m prettery sure add env plugins dir Log has INFO ```21 04 23 02 33 34 INFO org apache pinot spi plugin PluginManager Plugins root dir is 21 04 23 02 33 34 INFO org apache pinot spi plugin PluginManager Trying to load plugins pinot gcs ``` Full log ``` retrieving org apache spark spark submit parent adf0fd1c d000 4782 8499 d41f1396e726 confs default 0 artifacts copied 9 already retrieved 0kB 17ms 21 04 23 02 33 34 INFO org apache pinot spi plugin PluginManager Plugins root dir is 21 04 23 02 33 34 INFO org apache pinot spi plugin PluginManager Trying to load plugins pinot gcs 21 04 23 02 33 35 INFO org apache pinot spi ingestion batch IngestionJobLauncher SegmentGenerationJobSpec org apache pinot spi ingestion batch spec SegmentGenerationJobSpec authToken null cleanUpOutputDir false excludeFileNamePattern null executionFrameworkSpec extraConfigs stagingDir name spark segmentGenerationJobRunnerClassName org apache pinot plugin ingestion batch spark SparkSegmentGenerationJobRunner segmentMetadataPushJobRunnerClassName null segmentTarPushJobRunnerClassName org apache pinot plugin ingestion batch spark SparkSegmentTarPushJobRunner segmentUriPushJobRunnerClassName org apache pinot plugin ingestion batch spark SparkSegmentUriPushJobRunner failOnEmptySegment false includeFileNamePattern glob avro inputDirURI jobType SegmentCreationAndUriPush outputDirURI overwriteOutput true pinotClusterSpecs controllerURI pinotFSSpecs className org apache pinot plugin filesystem GcsPinotFS configs null scheme gs pushJobSpec pushAttempts 2 pushParallelism 2 pushRetryIntervalMillis 1000 segmentUriPrefix null segmentUriSuffix null recordReaderSpec className org apache pinot plugin inputformat avro AvroRecordReader configClassName null configs null dataFormat avro segmentCreationJobParallelism 0 segmentNameGeneratorSpec configs segment name prefix rule logs uat exclude sequence id true type simple tableSpec schemaURI tableConfigURI tableName RuleLogsUAT tlsSpec null```

Phúc Huỳnh · Answer

when i remove ` Dplugins include=pinot gcs` i found another jar files Maybe it s root cause issue

Xiang Fu · Answer

for Plugins root dir can you try absolute path

Phúc Huỳnh · Answer

let s me try

Phúc Huỳnh · Answer

hmm bz spark context working dir base on context id ``` tmp 2694644d46744db78cbe27e6dd833f2a``` so it s hard to get absolute path

Xiang Fu · Answer

hmm does `$ pwd plugins` work

Phúc Huỳnh · Answer

`$ pwd plugins` will be current dir on remote control machine Not worker exec machine

Xiang Fu · Answer

hmm

Xiang Fu · Answer

the ingestion job only tar gz plugin dir on driver and set it

Xiang Fu · Answer

the worker will untar the plugin dir from the context

Xiang Fu · Answer

this is how we package the plugin dir and add to sparkContext

Xiang Fu · Answer

then each worker will untar the plugin dir from the targz file

Xiang Fu · Answer

you can check the class `org apache pinot plugin ingestion batch spark SparkSegmentGenerationJobRunner`

Phúc Huỳnh · Answer

hmm i will try with option cluster initialization actions gt gsutil cp plugins dir to cluster folder

Xiang Fu · Answer

i see then it means the spark driver has no access to plugin dir

Xiang Fu · Answer

or you will download the plugin dir

Phúc Huỳnh · Answer

i will predownload the plugins dir in relative folder

Phúc Huỳnh · Answer

all steps almost done But the final step is SegmentUriPushJob is errors Left Tab Logging spark job Right Tab logging pinot controller

Phúc Huỳnh · Answer

`PinotFS for scheme gs has not been initialized` again disappointed

Phúc Huỳnh · Answer

oh i finded the issue i loaded jar `pinot batch ingestion spark` from release branch

Xiang Fu · Answer

so the filesystem is not init yet

Xiang Fu · Answer

I think you found the issue

Phúc Huỳnh · Answer

hmmm One more issue

Phúc Huỳnh · Answer

API `v2 segments` seem conflict vs spark sendSegmentUris Something null that s make API return internal server errors

Phúc Huỳnh · Answer

logging in spark job request ```Start sending table RuleLogsUAT segment URIs gs bucket data year=2020 RuleLogsUAT OFFLINE 18316 18627 0 tar gz to locations org apache pinot spi ingestion batch spec PinotClusterSpec 499782c3 Sending table RuleLogsUAT segment URI gs bucket data year=2020 RuleLogsUAT OFFLINE 18316 18627 0 tar gz to location https domain for Sending request https domain v2 segments to controller pinot controller 0 pinot controller headless analytics svc cluster local version Unknown Caught temporary exception while pushing table RuleLogsUAT segment uri gs bucket data year=2020 RuleLogsUAT OFFLINE 18316 18627 0 tar gz to https domain will retry Got error status code 500 Internal Server Error with reason Caught internal server exception while uploading segment while sending request https domain v2 segments to controller pinot controller 0 pinot controller headless analytics svc cluster local version Unknown at org apache pinot common utils FileUploadDownloadClient sendRequest FileUploadDownloadClient java 451 at org apache pinot common utils FileUploadDownloadClient sendSegmentUri FileUploadDownloadClient java 771 at org apache pinot segment local utils SegmentPushUtils lambda$sendSegmentUris$1 SegmentPushUtils java 178 at org apache pinot spi utils retry BaseRetryPolicy attempt BaseRetryPolicy java 50 at org apache pinot segment local utils SegmentPushUtils sendSegmentUris SegmentPushUtils java 175 at org apache pinot plugin ingestion batch spark SparkSegmentUriPushJobRunner$1 call SparkSegmentUriPushJobRunner java 127 at org apache pinot plugin ingestion batch spark SparkSegmentUriPushJobRunner$1 call SparkSegmentUriPushJobRunner java 117 at org apache spark api java JavaRDDLike $anonfun$foreach$1 JavaRDDLike scala 352 at org apache spark api java JavaRDDLike $anonfun$foreach$1$adapted JavaRDDLike scala 352 at scala collection Iterator foreach Iterator scala 943 at scala collection Iterator foreach$ Iterator scala 943 at org apache spark InterruptibleIterator foreach InterruptibleIterator scala 28 at org apache spark rdd RDD $anonfun$foreach$2 RDD scala 1012 at org apache spark rdd RDD $anonfun$foreach$2$adapted RDD scala 1012 at org apache spark SparkContext $anonfun$runJob$5 SparkContext scala 2242 at org apache spark scheduler ResultTask runTask ResultTask scala 90 at org apache spark scheduler Task run Task scala 131 at org apache spark executor Executor$TaskRunner $anonfun$run$3 Executor scala 497 at org apache spark util Utils$ tryWithSafeFinally Utils scala 1439 at org apache spark executor Executor$TaskRunner run Executor scala 500 at java util concurrent ThreadPoolExecutor runWorker ThreadPoolExecutor java 1149 at java util concurrent ThreadPoolExecutor$Worker run ThreadPoolExecutor java 624 at java lang Thread run Thread java 748 ```

Phúc Huỳnh · Answer

As far as i can tell Maybe headers request is null then default uploadType is `SEGMENT` gt exception when multiPart file null

Xiang Fu · Answer

hmm

Xiang Fu · Answer

for your pinot controller server do they have those pinot filesystem configured

Xiang Fu · Answer

also can you try segmentMetadataPush

Xiang Fu · Answer

```segmentMetadataPushJobRunnerClassName org apache pinot plugin ingestion batch spark SparkSegmentMetadataPushJobRunner jobType SegmentCreationAndMetadataPush```

Phúc Huỳnh · Answer

well i think i has found the problem In nginx ingress log i found ```4 519 client sent invalid header line DOWNLOAD URI <gs my bucket test data np data RuleLogsUAT OFFLINE 18117 18731 0 tar gz> while reading client request headers client 10 255 160 94 ```

Phúc Huỳnh · Answer

Phúc Huỳnh · Answer

```When the use of underscores is disabled request header fields whose names contain underscores are marked as invalid and become subject to the ignore invalid headers directive ```

Phúc Huỳnh · Answer

I think nginx ingress remove all headers which are the invalid name

Xiang Fu · Answer

oic

Xiang Fu · Answer

then you need to change nginx to keep those headers