How to ingest Data into pinot on kubernetes using native batch ingestion? Hi, I am trying to ingest csv data into pinot deployed on kubernetes using LaunchDataIngestionJob arg. I have verified that the table has been created on pinot and the job-spec and csv data are present on the node. This is my job-spec file

apiVersion: batch/v1
kind: Job
metadata:
  name: pinot-case-offline-ingestion
  namespace: my-pinot-kube
spec:
  template:
    spec:
      containers:
        - name: pinot-load-case-offline
          image: apachepinot/pinot:0.3.0-SNAPSHOT
          args: ["LaunchDataIngestionJob", "-jobSpecFile", "/opt/data/table-configs/case_history/job-spec.yml"]
          volumeMounts:
            - name: mount-data
              mountPath: /opt/data
      restartPolicy: OnFailure
      volumes:
        - name: mount-data
          hostPath:
            path: /opt/data
  backoffLimit: 100

After applying this job to node, nothing happens and this is the log of the pod.

SegmentGenerationJobSpec: 
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**/*.csv
inputDirURI: /opt/data/csv_data/case_prod_data
jobType: SegmentCreationAndTarPush
outputDirURI: /pinot-segments/case_history
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: '<http://192.168.49.2:30892/>'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: null
recordReaderSpec:
  className: org.apache.pinot.plugin.inputformat.csv.CSVRecordReader
  configClassName: org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig
  configs: {delimiter: '|', multiValueDelimiter: ''}
  dataFormat: csv
segmentNameGeneratorSpec:
  configs: {segment.name.prefix: case_history, exclude.sequence.id: 'true'}
  type: normalizedDate
tableSpec: {schemaURI: null, tableConfigURI: null, tableName: case_history}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS

Am I ingesting the data incorrectly ?

I think you are missing pushJobSpec?

Hi @Xiang Fu, Thank you for helping. I tried adding pushJobSpec to job-spec

pushJobSpec:
  pushParallelism: 2
  pushAttempts: 2
  pushRetryIntervalMillis: 1000

But the job gets completed with no errors. And the pod log is

SegmentGenerationJobSpec: 
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**/*.csv
inputDirURI: /opt/data/csv_data/case_prod_data
jobType: SegmentCreationAndTarPush
outputDirURI: /pinot-segments/case_history
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: '<http://192.168.49.2:30892/>'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: {pushAttempts: 2, pushParallelism: 2, pushRetryIntervalMillis: 1000,
  segmentUriPrefix: null, segmentUriSuffix: null}
recordReaderSpec:
  className: org.apache.pinot.plugin.inputformat.csv.CSVRecordReader
  configClassName: org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig
  configs: {delimiter: '|', multiValueDelimiter: ''}
  dataFormat: csv
segmentNameGeneratorSpec:
  configs: {segment.name.prefix: case_history, exclude.sequence.id: 'true'}
  type: normalizedDate
tableSpec: {schemaURI: null, tableConfigURI: null, tableName: case_history}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS

ok

Here is the log of the job:

16:26:48:ayush@:pinot :alien: kubectl -n my-pinot-kube describe jobs.batch pinot-case-offline-ingestion 
Name:           pinot-case-offline-ingestion
Namespace:      my-pinot-kube
Selector:       controller-uid=25b4e843-b600-4de2-a2ad-584ac8ce17b5
Labels:         controller-uid=25b4e843-b600-4de2-a2ad-584ac8ce17b5
                job-name=pinot-case-offline-ingestion
Annotations:    <none>
Parallelism:    1
Completions:    1
Start Time:     Fri, 05 Mar 2021 16:26:41 -0500
Completed At:   Fri, 05 Mar 2021 16:26:44 -0500
Duration:       3s
Pods Statuses:  0 Running / 1 Succeeded / 0 Failed
Pod Template:
  Labels:  controller-uid=25b4e843-b600-4de2-a2ad-584ac8ce17b5
           job-name=pinot-case-offline-ingestion
  Containers:
   pinot-load-case-offline:
    Image:      apachepinot/pinot:0.3.0-SNAPSHOT
    Port:       <none>
    Host Port:  <none>
    Args:
      LaunchDataIngestionJob
      -jobSpecFile
      /opt/data/table-configs/case_history/job-spec.yml
    Environment:  <none>
    Mounts:
      /opt/data from mount-data (rw)
  Volumes:
   mount-data:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/data
    HostPathType:  
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  27s   job-controller  Created pod: pinot-case-offline-ingestion-mfvrx
  Normal  Completed         24s   job-controller  Job completed

The following is the job spec file to refer. What should be the pinotClusterSpecs.controllerURI value? I tried changing it to anything gibberish and I faced the same logs. I think, my value of pinotClusterSpecs.controllerURI is incorrect.

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '/opt/data/csv_data/case_prod_data'
includeFileNamePattern: 'glob:**/*.csv'
outputDirURI: '/pinot-segments/case_history'
overwriteOutput: true
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
  dataFormat: 'csv'
  className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
  configs:
    delimiter: '|'
    multiValueDelimiter: ''
tableSpec:
  tableName: 'case_history'
pinotClusterSpecs:
  # - controllerURI: 'pinot-controller:9000'
  - controllerURI: '<http://192.168.49.2:30892/>'
segmentNameGeneratorSpec:
  type: normalizedDate
  configs:
    segment.name.prefix: 'case_history'
    exclude.sequence.id: true
pushJobSpec:
  pushParallelism: 2
  pushAttempts: 2
  pushRetryIntervalMillis: 1000

then are there data on

/opt/data/csv_data/case_prod_data

yes. I checked by running a ubuntu container and bashed into it. there is data present on this path

can you try a newer image as well

ok. so changed the image. it worked. at the very end of the log it says

Response for pushing table case_history segment case_history to location <http://192.168.49.2:30892> - 200: {"status":"Successfully uploaded segment: case_history of table: case_history"}

hmm

seems, like another issue that I have to look into. But anyways, thank you very much @Xiang Fu for you promt responses and help. The new image worked out well.

can you check pinot server log?

ok. I do see some errors on pinot-server.

hmm when you start pinot server did you give a clustername?

I start pinot using helm like

kubectl create ns my-pinot-kube
helm install pinot /home/ayush/spyne/incubator-pinot/kubernetes/helm/pinot -n my-pinot-kube --set replicas=1

hmmm

ok. All the pinot workers are in running state. I do see these 2 errors on pinot-controller

WARN [PinotInstanceRestletResource] [grizzly-http-server-1] Admin port is not set for instance: Server_pinot-server-0.pinot-server-headless.my-pinot-kube.svc.cluster.local_8098
...
...

WARN [PinotInstanceRestletResource] [grizzly-http-server-1] Grpc port is not set for instance: Controller_pinot-controller-0.pinot-controller-headless.my-pinot-kube.svc.cluster.local_9000
...
...

this means your controller is up, but no pinot server is connected to the cluster

yes. restarting the node

cool!

yes. whenever I start using helm, zookeeper and controller are the last ones to start and because of that server and broker takes multiple restarts.