Hi team, I am trying to ingest data in pinot using Minion based SegmentGenerationAndPushTask. I set both my task config and ingestion config to use "METADATA" push mode. But when I look at the Minion logs, it still shows "push mode TAR" instead of "push mode METADATA". Is metadata push supported with this task type?

METADATA push mode requires deep-store because in that mode the segment is copied to deep-store, and the uri + metadata is pushed to the controller. In case you are using local file system, then there is no uri to pass. cc: @Manish

I am using S3 as the deep store and my input source is also S3. Could you please verify once if anything is wrong in this config? Thanks!

"task": {
      "taskTypeConfigsMap": {
        "SegmentGenerationAndPushTask": {
          "schedule": "0 0 2 * * ?",
          "inputDirURI": "<s3://data-pinot-dev/pinot/input/>",
          "includeFileNamePattern": "glob:**/databackfill",
          "excludeFileNamePattern": "glob:**/_*",
          "inputFormat": "parquet",
          "push.mode": "METADATA",
          "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
          "input.fs.prop.region": "ap-south-1",
          "segmentNameGenerator.type": "normalizedDate",
          "segment.name.prefix": "req_lost_metadata_push",
          "deepStoreType": "S3",
          "deepStoreURI": "<s3://data-pinot-dev/pinot/segments/>",
          "segment.name.generator.type": "normalizedDate",
          "segment.name.generator.configs.segmentNamePrefix": "req_lost_metadata_push",
          "segment.name.generator.configs.appendUUIDToSegmentName": "false",
          "segmentNameGenerator": "normalizedDate"
        }
      }
    },
    "ingestionConfig": {
      "batchIngestionConfig": {
        "batchConfigMaps": [
          {
            "inputDirURI": "<s3://data-pinot-dev/pinot/input/>",
            "includeFileNamePattern": "glob:**/databackfill",
            "excludeFileNamePattern": "glob:**/_*",
            "inputFormat": "parquet",
            "push.mode": "METADATA",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader",
            "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
            "input.fs.prop.region": "ap-south-1",
            "segmentNameGenerator.type": "normalizedDate",
            "segment.name.prefix": "req_lost_metadata_push",
            "deepStoreType": "S3",
            "deepStoreURI": "<s3://data-pinot-dev/pinot/segments/>"
          }
        ],
        "segmentIngestionType": "APPEND",
        "segmentIngestionFrequency": "DAILY",
        "consistentDataPush": false
      },
      "continueOnError": false,
      "rowTimeValueCheck": false,
      "segmentTimeValueCheck": true
    },
    "isDimTable": false
  }
}

cc @Shounak Kulkarni

can you check for the log line

Executing SegmentGenerationAndPushTask with task config:

and share what follows?

Checking with outputDirURI , thanks!