Hello I am trying to get started with a batch offline ingest Apache Pinot #troubleshooting

Hello... I am trying to get started with a batch/o...

Saoirse Amarteifio

09/27/2021, 12:18 PM

Hello... I am trying to get started with a batch/offline ingestion

Copy code

../apache-pinot-0.8.0-bin/bin/pinot-ingestion-job.sh -jobSpecFile ./pinot_ingest_samples/batch_ingestion_no_comment.yaml

Using this file

Copy code

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'
jobType: SegmentCreationAndUriPush
inputDirURI: '<s3://data-platform/samples/data/>'
includeFileNamePattern: 'glob:**/*.parquet'
outputDirURI: '<s3://pinot-development/sample/segments>'
overwriteOutput: true
pinotFSSpecs:
  - scheme: s3
    className: org.apache.pinot.plugin.filesystem.S3PinotFS
    configs:
      region: 'us-east-1'
recordReaderSpec:
  dataFormat: 'parquet'
  className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader'
tableSpec:
  tableName: 'sample'
  schemaURI: '<http://pinot-controller.pinot.cluster.svc.local:9000/tables/sample/schema>'
  tableConfigURI: '<http://pinot-controller.pinot.cluster.svc.local:9000/tables/sample>'
pinotClusterSpecs:
  - controllerURI: '<http://pinot-controller.pinot.cluster.svc.local:9000>'
pushJobSpec:
  pushAttempts: 2
  pushRetryIntervalMillis: 1000
  segmentUriPrefix: '<s3://pinot-development>'
  segmentUriSuffix: ''

But i get a class not found exception

java.lang.RuntimeException: Failed to create IngestionJobRunner instance for class - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner

Saoirse Amarteifio

09/27/2021, 12:20 PM

I am wondering if there are some Java nooob things i need to do like setting certain env vars, classpaths or something. I expected to just download the binary for pinot, run the script with an input file, and for it to just work but not so.

Saoirse Amarteifio

09/27/2021, 12:24 PM

... or maybe i should use docker locally for testing as well as when deploying agents to K8s and it will make life easier - i can do that too?

Richard Startin

09/27/2021, 12:27 PM

which JDK version are you using?

Richard Startin

09/27/2021, 12:28 PM

there was a regression in 0.8.0 which has been fixed by @Ken Krugler which might relate to this

Saoirse Amarteifio

09/27/2021, 12:31 PM

My java version output is

Copy code

java version "11.0.12" 2021-07-20 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.12+8-LTS-237)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.12+8-LTS-237, mixed mode)

Richard Startin

09/27/2021, 12:32 PM

thanks, I'm not sure the fix applies then.

Saoirse Amarteifio

09/27/2021, 12:33 PM

Ok thanks Richard. Ill try the same thing in docker later - running locally is not something i really need to do except to get that warm fuzzy feeling when it works for the first time

Saoirse Amarteifio

09/27/2021, 12:39 PM

Trailing question; is there another way for me to use the REST interface on the controller to ingest data. Really all i want is some scheduled way to move data from parquet files on S3 to table segments ? I guess this is pretty basic.

Ken Krugler

09/28/2021, 1:37 AM

Hi @Saoirse Amarteifio - I’d suggest asking that question on the #CDRCA57FC channel.

Saoirse Amarteifio

09/28/2021, 11:17 AM

Thanks @Ken Krugler for the guidance - will do!

2 Views

Open in Slack

Previous Next