Question on importing partitioned parquet data from S3… I have looked at the docs, examples, and searched Slack, but did not find an answer. I have a synthetic data set based on the transcript data example from the docs. That data is produced in Spark.

transcripts_bucket = "<s3://transcript-parquet/1/>"
transcript_df.write.mode("overwrite").parquet(transcripts_bucket)

I run a Spark job to transform into segments and import the data into Pinot

spark_args = {
    'master': '<spark://spark-master:7077>',
    'deploy_mode': 'cluster',
    'name': 'segments-from-parquet',
    'class': 'org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand',
    'executor_memory': '2G',
    'executor_cores': '1',
    'total_executor_cores': '2',
    'verbose': True,
    'conf': [f"spark.driver.extraJavaOptions='{EXTRA_JAVA_OPTIONS}'"],
    'main_file_args': '-jobSpecFile=/home/job_specs/transcript_job_spec.yml'
    }

main_file = f'{PINOT_DISTRIBUTION_DIR}/lib/pinot-all-{PINOT_VERSION}-jar-with-dependencies.jar'

app = SparkJob(main_file, **spark_args)
app.submit()

All columns populate as expected. Schema is

{
  "schemaName": "transcript_indexed",
  "dimensionFieldSpecs": [
    {
      "name": "studentID",
      "dataType": "INT"
    },
    {
      "name": "firstName",
      "dataType": "STRING"
    },
    {
      "name": "lastName",
      "dataType": "STRING"
    },
    {
      "name": "gender",
      "dataType": "STRING"
    },
    {
      "name": "subject",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "score",
      "dataType": "FLOAT"
    }
  ],
  "timeFieldSpec": {
    "incomingGranularitySpec": {
      "name": "examTime",
      "dataType": "LONG",
      "timeType": "MILLISECONDS"
    }
  }
}

If I partition my data by subject when I write the parquet files, I have an unexpected outcome, the subject field in the Pinot segments is null.

transcripts_bucket = "<s3://transcript-parquet/1/>"

transcript_df.write.mode("overwrite").partitionBy("subject").parquet(transcripts_bucket)

Are we thinking about this incorrectly…. in a way that Pinot does not support? Alternately, is there some change we need to make in the configs to work with data lake data that is normally partitioned? The data on Minio — my S3 service fake looks like this once partitioned by subject

fwiw #C011C9JHN7R might be a bit better of a channel, but what's your table setup like?

Hi Ashwin. Table config

{
  "OFFLINE": {
    "tableName": "transcript_indexed_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "replication": "1",
      "timeType": "MILLISECONDS",
      "schemaName": "transcript_indexed",
      "timeColumnName": "examTime",
      "minimizeDataMovement": false
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [],
      "rangeIndexVersion": 2,
      "loadMode": "MMAP",
      "enableDefaultStarTree": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": false,
      "autoGeneratedInvertedIndex": false,
      "enableDynamicStarTreeCreation": false,
      "optimizeDictionaryForMetrics": false,
      "noDictionarySizeRatioThreshold": 0,
      "createInvertedIndexDuringSegmentGeneration": false
    },
    "metadata": {},
    "isDimTable": false
  }
}

and what's your job spec file?

transcript_job_spec.yml

@Steven Hall one thing is that you dont have

stagingDir

set?

yes

sorry, yes you have it set somewhere else, or yes you still need to set it?

OK, tell me more. It seems to work, until partitioned in a specific manner. I can import 250K records, until I change the partitioning. Let me look at staging dir. Do you have a doc link I should look at?

I think your last comment likely gets to the root of the issue. I have been thinking about this wrong. It should be partitioned by time for Pinot. Thanks, let me investigate that feedback a bit. Appreciate the help.

yeah, and specifically by time range is ideal

👍