Hey guys, I am stuck at ingesting some batches of data into pinot, for some reason every time we run the ingestion job, although I see the tar files in the destination folder inside the deep store (about 500 tar.gz files), I can only see one segment in zookeeper's metadata, and it looks like that single segment is the only segment queryable in our offline table, I will share our specs and the command I am running to start the ingestion so that you can tell me please if I am missing something.

# executionFrameworkSpec: Defines ingestion jobs to be running.
executionFrameworkSpec:

  # name: execution framework name
  name: 'standalone'
  
  # segmentGenerationJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentGenerationJobRunner interface.
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  
  # segmentTarPushJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentTarPushJobRunner interface.
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  
  # segmentUriPushJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentUriPushJobRunner interface.
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
  
  # segmentMetadataPushJobRunnerClassName: class name implements org.apache.pinot.plugin.ingestion.batch.common.SegmentMetadataPushJobRunner interface.
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'

# jobType: Pinot ingestion job type.
# Supported job types are:
#   'SegmentCreation'
#   'SegmentTarPush'
#   'SegmentUriPush'
#   'SegmentCreationAndTarPush'
#   'SegmentCreationAndUriPush'
# Recommended to set jobType to SegmentCreationAndMetadataPush for production environment where Pinot Deep Store is configured  
jobType: SegmentCreationAndMetadataPush

# inputDirURI: Root directory of input data, expected to have scheme configured in PinotFS.
inputDirURI: 'gs://$BUCKET_NAME/bigquery-loads/'

# includeFileNamePattern: include file name pattern, supported glob pattern.
# Sample usage:
#   'glob:*.avro' will include all avro files just under the inputDirURI, not sub directories;
#   'glob:**\/*.avro' will include all the avro files under inputDirURI recursively.
includeFileNamePattern: 'glob:**/*.avro'

# excludeFileNamePattern: exclude file name pattern, supported glob pattern.
# Sample usage:
#   'glob:*.avro' will exclude all avro files just under the inputDirURI, not sub directories;
#   'glob:**\/*.avro' will exclude all the avro files under inputDirURI recursively.
# _excludeFileNamePattern: ''

# outputDirURI: Root directory of output segments, expected to have scheme configured in PinotFS.
outputDirURI: 'gs://$BUCKET_NAME/controller/test/my_table/offline/'

# overwriteOutput: Overwrite output segments if existed.
overwriteOutput: true

# Job parallelism for segment creation
segmentCreationJobParallelism: 4

# pinotFSSpecs: defines all related Pinot file systems.
pinotFSSpecs:

  - # scheme: used to identify a PinotFS.
    # E.g. local, hdfs, dbfs, gs, etc
    scheme: gs
    # className: Class name used to create the PinotFS instance.
    # E.g.
    #   org.apache.pinot.spi.filesystem.LocalPinotFS is used for local filesystem
    #   org.apache.pinot.plugin.filesystem.AzurePinotFS is used for Azure Data Lake
    #   org.apache.pinot.plugin.filesystem.HadoopPinotFS is used for HDFS
    #   org.apache.pinot.plugin.filesystem.GcsPinotFS is used for GCS
    className: org.apache.pinot.plugin.filesystem.GcsPinotFS
    configs:
      projectId: '$GOOGLE_PROJECT_NAME'
      gcpKey: '/opt/pinot/deployment/gcs/key.json'

# recordReaderSpec: defines all record reader
recordReaderSpec:

  # dataFormat: Record data format, e.g. 'avro', 'parquet', 'orc', 'csv', 'json', 'thrift' etc.
  dataFormat: 'avro'
  
  # className: Corresponding RecordReader class name.
  # E.g.
  #   org.apache.pinot.plugin.inputformat.avro.AvroRecordReader
  #   org.apache.pinot.plugin.inputformat.csv.CSVRecordReader
  #   org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader
  #   org.apache.pinot.plugin.inputformat.json.JSONRecordReader
  #   org.apache.pinot.plugin.inputformat.orc.ORCRecordReader
  #   org.apache.pinot.plugin.inputformat.thrift.ThriftRecordReader
  className: 'org.apache.pinot.plugin.inputformat.avro.AvroRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.avro.AvroRecordReaderConfig'

# tableSpec: defines table name and where to fetch corresponding table config and table schema.
tableSpec:

  # tableName: Table name
  tableName: 'my_table'

  # schemaURI: defines where to read the table schema, supports PinotFS or HTTP.
  # E.g.
  #   <hdfs://path/to/table_schema.json>
  #   <http://localhost:9000/tables/myTable/schema>
  schemaURI: '<http://pinot-controller:9000/tables/my_table/schema>'

  # tableConfigURI: defines where to reade the table config.
  # Supports using PinotFS or HTTP.
  # E.g.
  #   <hdfs://path/to/table_config.json>
  #   <http://localhost:9000/tables/myTable>
  # Note that the API to read Pinot table config directly from pinot controller contains a JSON wrapper.
  # The real table config is the object under the field 'OFFLINE'.
  tableConfigURI: '<http://pinot-controller:9000/tables/my_table>'

# pinotClusterSpecs: defines the Pinot Cluster Access Point.
pinotClusterSpecs:

  - # controllerURI: used to fetch table/schema information and data push.
    # E.g. <http://localhost:9000>
    controllerURI: '<http://pinot-controller:9000>'

pushJobSpec:

  # Job parallelism for segment push
  pushParallelism: 4

  # pushAttempts: number of attempts for push job, default is 1, which means no retry.
  pushAttempts: 5

  # pushRetryIntervalMillis: retry wait Ms, default to 1 second.
  pushRetryIntervalMillis: 30000
  
  # Applicable for URI and METADATA push types.
  #If true, and if segment was not already in the deep store, move it to deep store.
  copyToDeepStoreForMetadataPush: true

This is the command

./bin/pinot-admin.sh LaunchDataIngestionJob \
  -jobSpecFile deployment/ingestion/my_table_ingestion_job.yaml

I don't see any errors either in the job's pod nor in the controller or any minions, any insight would be really appreciated. Thanks!