Hi All, NEED HELP !! Im trying to ingest CSV file ...
# troubleshooting
b
Hi All, NEED HELP !! Im trying to ingest CSV file which is delimited by tab space, but Im getting only null value populated with no error in ingestion process Below is the ingestion job spec and sample data and screenshot of data populated in pinot INGESTION JOB SPEC
Copy code
executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '/input_data/'
includeFileNamePattern: 'glob:**/*.csv'
outputDirURI: '/output_segments/'
overwriteOutput: true
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
  dataFormat: 'csv'
  className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
  configs:
    multiValueDelimiter: '\$'
    delimiter: '	'
tableSpec:
  tableName: 'asset_control_family'
pinotClusterSpecs:
  - controllerURI: 'http://localhost:9000'
pushJobSpec:
  pushAttempts: 2
  pushRetryIntervalMillis: 1000
DATA.CSV
Copy code
App ID	App Name
100008	MKind_Xcde_-_Cox
100008	MKind_Xcde_-_Cox
100008	MKind_Xcde_-_Cox
100008	MKind_Xcde_-_Cox
100008	MKind_Xcde_-_Cox
Just got it, name in the schema is not same as CSV header Is there a way to map the column header with columns in schema?
x
You may need to create your own csv decoder yourself or just make transform functions for those
b
Got it, so there is no implicit way within pinot to do the mapping. it should be done before dropping the CSV Thanks.
x
Right, or you just copy the csv and fix the header