Apache Pinot

Job_spec.yaml

image.png

Hi All, I am new to Pinot, we are trying to load csv file into table using config.yaml file. but we are getting error. could you please help us to load the file into table.

```executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'
# Recommended to set jobType to SegmentCreationAndMetadataPush for production environment where Pinot Deep Store is configured  
jobType: SegmentCreationAndTarPush
inputDirURI: 'tmp/pinot_input/'
includeFileNamePattern: 'glob:**/transcrpit.csv'
outputDirURI: '/tmp/pinot_output/'
overwriteOutput: true
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
  dataFormat: 'csv'
  className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
  configs:
    fileFormat: 'default' #should be one of default, rfc4180, excel, tdf, mysql
    header: 'columnName seperated by delimiter'
    delimiter: '|'
    multiValueDelimiter: '-'
tableSpec:
  tableName: 'transcript'
pinotClusterSpecs:
  - controllerURI: '<http://local:9000>'
pushJobSpec:
  pushAttempts: 2
  pushRetryIntervalMillis: 1000```
Your `configs` section needs to be indented to go under `recordReaderSpec`

`controllerURI` might be incorrect as well? I think that should probably be `<http://localhost:9000>`

Job_spec_yaml.pas

error_log.txt

Thank you for the reply. i made the changes in the yaml file. when i run yaml, table became bad state. could you please tell, what i am missing. <@U01DS9A0DS4>

{
  "tableName": "transcript",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "replication": 1,
    "timeColumnName": "timestampInEpoch",
    "timeType": "MILLISECONDS",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": 365
  },
  "tenants": {
    "broker":"DefaultTenant",
    "server":"DefaultTenant"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP"
  },
  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "DAILY"
    }
  },
  "metadata": {}
} <@U01DS9A0DS4>

Hi Team, when everytime i run the job, my table become bad. could you please help me to overcome from this issue. <@UDQU92KBK> <@U01DS9A0DS4>

In the code the error seems to be that it's unable to create a directory for the table

so I wonder maybe the permissions are missing or something?

I think it's the `dataDir` for the Pinot Server. Pinot isn't able to create that directory from what I can tell

Also please double check against external view in ZK browser on the status of the segment. 

<@U01DS9A0DS4> <@UDQU92KBK> It is working fine now. i think an error occurred due to permission