Saoirse Amarteifio
10/11/2021, 5:12 PMRecordReader initialized will read a total of 8 records.
at row 0. reading next block
block read in memory in 1 ms. row count = 8
Start building IndexCreator!
Finished records indexing in IndexCreator!
Failed to generate Pinot segment for file - <s3://bucket/samples/data/myData/test.parquet>
java.lang.IllegalArgumentException: null
at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) ~[pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-11f8550b9b2881ede4d105416ed970a5dd708463]
at org.apache.pinot.segment.spi.creator.name.SimpleSegmentNameGenerator.generateSegmentName(SimpleSegmentNameGenerator.java:53) ~[pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-11f8550b9b2881ede
4d105416ed970a5dd708463]
Can anyone suggest what illegal thing i am doing from this error message? adding jobSpec in thread...Saoirse Amarteifio
10/11/2021, 5:14 PMexecutionFrameworkSpec:
name: 'standalone'
segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'
jobType: SegmentCreationAndUriPush
inputDirURI: 's3://...'
includeFileNamePattern: 'glob:**/*.parquet'
outputDirURI: 's3://...'
overwriteOutput: true
pinotFSSpecs:
- scheme: s3
className: org.apache.pinot.plugin.filesystem.S3PinotFS
configs:
region: 'us-east-1'
recordReaderSpec:
dataFormat: 'parquet'
className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader'
tableSpec:
tableName: 'MY_TABLE'
schemaURI: '<http://pinot-controller.pinot.svc.cluster.local:9000/tables/MY_TABLE/schema>'
tableConfigURI: '<http://pinot-controller.pinot.svc.cluster.local:9000/tables/MY_TABLE>'
pinotClusterSpecs:
- controllerURI: '<http://pinot-controller.pinot.svc.cluster.local:9000>'
pushJobSpec:
pushAttempts: 2
pushRetryIntervalMillis: 1000
Saoirse Amarteifio
10/11/2021, 5:53 PMsegmentNameGeneratorSpec
I realize its helpful to scan through the logs above the error and observe where some parameters are null and sometimes it matters!