Hi @Xiang Fu @Xiaobing, I have a trouble in load data from s3 to pinot offline table. Sharing tableconfig and segment which is created

{
  "OFFLINE": {
    "tableName": "test_transcript_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "schemaName": "test_transcript",
      "replication": "1",
      "timeColumnName": "timestamp",
      "segmentPushFrequency": "HOURLY",
      "segmentPushType": "APPEND",
      "replicasPerPartition": "1"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [],
      "noDictionaryColumns": [],
      "rangeIndexColumns": [],
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [],
      "bloomFilterColumns": [],
      "loadMode": "MMAP",
      "onHeapDictionaryColumns": [],
      "varLengthDictionaryColumns": [],
      "enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": false
    },
    "metadata": {},
    "quota": {},
    "task": {
      "taskTypeConfigsMap": {
        "SegmentGenerationAndPushTask": {
          "schedule": "/5 * * * * ?",
          "tableMaxNumTasks": "10"
        }
      }
    },
    "routing": {},
    "query": {},
    "ingestionConfig": {
      "batchIngestionConfig": {
        "batchConfigMaps": [
          {
            "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
            "input.fs.prop.region": "us-east-1",
            "input.fs.prop.secretKey": "*****",
            "input.fs.prop.accessKey": "*****",
            "inputDirURI": "<s3://pp-airflow-qa/dremio_test_files/jsonfiles/>",
            "includeFileNamePattern": "glob:**/*.json",
            "excludeFileNamePattern": "glob:**/*.tmp",
            "inputFormat": "json"
          }
        ],
        "segmentIngestionType": "APPEND",
        "segmentIngestionFrequency": "HOURLY"
      }
    },
    "isDimTable": false
  }
}

as if a segment was a file

Can you mask the s3 keys from this properties so that they are not visible to others. Secondly, in the one segment, how many records are there?

12 records

Ok. And you many files and records per file are present in the inputDirURI directory ?

3 files each with 12 records in it

Ok. and each of these files contain the exact same record?

no different data

yep, that's the problem

okay

"segmentNameGenerator.type": "inputFile",    
  "segmentNameGenerator.configs.file.path.pattern": ".",
  "segmentNameGenerator.configs.segment.name.template": "${filePathPattern:\\1}"

Add this inside batchConfigMaps

sure I"ll make changes