Hi all, I have some parquet part files in GCS that...
# troubleshooting
j
Hi all, I have some parquet part files in GCS that I want to import into an OFFLINE table that already uses GCS (same bucket) as its deep store. I think I'm configuring the job spec correctly but I keep getting this error related to Kerberos and hadoop, which I didn't think were supposed to be involved in this process at all. I'm running 1.2.0 and I'll share more details in đź§µ
KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
m
What's the PinotFS that you set?
j
This is our job config:
Copy code
executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'
jobType: SegmentCreationAndMetadataPush
inputDirURI: '<gs://etsy-pinot-data/backfill/keyword_research_search_data/2025/07/23>'
includeFileNamePattern: 'glob:**/*.parquet'
outputDirURI: '<gs://etsy-pinot-data/keyword_research_search_data/backfilled>'
# overwriteOutput: Overwrite output segments if existed.
overwriteOutput: true
pinotFSSpecs:
  - scheme: gs
    className: org.apache.pinot.plugin.filesystem.GcsPinotFS
    configs:
      projectId: 'etsy-syseng-gke-sandbox'
      gcpKey: '/tmp/pinot-gcp-key.json'
tableSpec:
  tableName: 'keyword_research_search_data_OFFLINE'
  #schemaURI: '<http://pinot-dev.pinot-controller.c.etsy-syseng-gke-prod.etsycloud.com/schemas/keyword_research_search_data>'
  schemaURI: '<http://pinot-controller.syseng.etsy-syseng-gke-sandbox.etsycloud.com/schemas/keyword_research_search_data>'
  tableConfigURI: '<http://pinot-controller.syseng.etsy-syseng-gke-sandbox.etsycloud.com/tables/keyword_research_search_data_OFFLINE>'
segmentNameGeneratorSpec:
  type: simple
pinotClusterSpecs:
  - controllerURI: '<http://pinot-controller.syseng.etsy-syseng-gke-sandbox.etsycloud.com>'
pushJobSpec:
  # pushAttempts: number of attempts for push job, default is 1, which means no retry.
  # pushAttempts: 3
  # pushParallelism: 2
  # pushRetryIntervalMillis: retry wait Ms, default to 1 second.
  # pushRetryIntervalMillis: 1000
  segmentUriPrefix: 'gs://'
  segmentUriSuffix: 'etsy-pinot-data/keyword_research_search_data/backfilled'
  pushFileNamePattern: 'glob:*'
I've tried using the
ingestFromURI
endpoint pointing at one file and get the same error
oh, it might be worth saying that we're running in kubernetes and generally have many tables pulling/pushing to GCS, but we don't usually do batch ingestion and this is our first time trying since we upgraded to 1.2.0 a few months ago
and I'm definitely not sure this is part of the problem, but I'm running the
pinot-admin.sh LaunchDataIngestionJob
command from a k8 server pod and
whoami
from the container outputs this
Copy code
whoami: cannot find name for user ID 1000
m
I do see you have set GcsPinotFS, unclear why it is looking for Kerberos.
j
this thread seems related and was also for 1.2.0 but the user was trying to use hdfs (just not kerberos) https://apache-pinot.slack.com/archives/C011C9JHN7R/p1745488950669169?thread_ts=1745177277.188329&amp;cid=C011C9JHN7R
update: I re-exported to avro and used avro in the job spec and it seems to have worked this time
so it's something to do with parquet I guess?
m
That still doesn’t make sense
j
oh yes, still not expected behavior. From the source, it looks like many of the parquet reader code paths (example) assume it's using the hdfs filesystem
maybe if I tried specifying
useParquetAvroRecordReader
or
useParquetNativeRecordReader
I could have skipped this condition that ends up hitting hadoop code
m
Try specifying RecordReaderSpec