https://pinot.apache.org/ logo
Join Slack
Powered by
Hi, having a similar but different issue with the ...
# troubleshooting
t
Hi, having a similar but different issue with the ‘standalone’ BatchIngestionJob, where ultimately the pod runs out of ephemeral storage (requires downloading more than 10G of data: 
The node was low on resource: ephemeral-storage. Container pinot-job-batch-ingestion was using 1944724Ki, which exceeds its request of 0.
). I have actually mounted a persistent volume to the pod executing this job, but it does not seem to be using it. I am currently mounting it at 
/var/pinot/minion/data
and 
/var/pinot/server/data
but this is not working for either. What directory should this volume be mounted to so that the BatchIngestJob uses the volume instead of the ephemeral storage? As a secondary question, is there a simpler way to do this within the kubernetes cluster running Pinot? Or is the standard way to utilize an external Spark Cluster with the custom compiled pinot image?
Here is my kubernetes job definition for refrence
Copy code
apiVersion: v1
kind: ConfigMap
metadata:
  name: batch-job-metadata-configmap
  namespace: datalake
data:
  batch-ingest-job-spec.yaml: |-
    executionFrameworkSpec:
        name: 'standalone'
        segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
        segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
        segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
    jobType: SegmentCreationAndTarPush
    inputDirURI: 's3://<omitted>'
    outputDirURI: 's3://<omitted>'
    overwriteOutput: true
    pinotFSSpecs:
        - scheme: s3
          className: org.apache.pinot.plugin.filesystem.S3PinotFS
          configs:
            region: '<omitted>'
    recordReaderSpec:
      dataFormat: 'parquet'
      className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader'
    tableSpec:
        tableName: 'uplinkpayloadevent'
    pinotClusterSpecs:
        - controllerURI: '<omitted>'
    pushJobSpec:
      pushParallelism: 2
      pushAttempts: 2
      pushRetryIntervalMillis: 1000
      segmentUriPrefix : 's3://'
      segmentUriSuffix : pinot-offline/
---
apiVersion: batch/v1
kind: Job
metadata:
  name: pinot-batch-ingest-job
  namespace: <omitted>
spec:
  template:
    spec:
      containers:
        - name: pinot-batch-ingestion
          image: apachepinot/pinot:latest
          args:
            - "LaunchDataIngestionJob"
            - "-jobSpecFile"
            - "/var/linklabs/batch/batch-ingest-job-spec.yaml"
          env:
            - name: JAVA_OPTS
              value: "-Xms32G -Xmx64G -Dpinot.admin.system.exit=true"
          resources:
            requests:
              memory: "32Gi"
            limits:
              memory: "64Gi"
          envFrom:
            - secretRef:
                name: <omitted>
          volumeMounts:
            - name: batch-job-metadata
              mountPath: /var/linklabs/batch
            - name: data
              mountPath: /var/pinot/server/data
      restartPolicy: OnFailure
      volumes:
        - name: batch-job-metadata
          configMap:
            name: batch-job-metadata-configmap
        - name: data
          persistentVolumeClaim:
            claimName: task-pv-pinot-etl-claim
  backoffLimit: 10
m
@Haitao Zhang ^^
h
ack
m
Also cc: @Seunghyun
k
The SegmentGenerationJobRunner code for standalone writes temp files to the directory returned by 
System._getProperty_("<http://java.io|java.io>.tmpdir")
. So if you can set that property to your permanent volume (or maybe a /tmp dir on that volume) then I think it would work as you want.
h
I would agree with this root cause. Maybe another way of fixing the problem is to mount the volumn to that dir
t
It does seem like mounting it on the 
/tmp
dir is working, the job is still running but it hasn’t gotten this far before yet - so that is a good sign!
Open in Slack
PreviousNext
Powered by