Apache Pinot

Hi, I'm trying to run the RealtimeToOfflineSegmentsTask on a minion. I'm using S3 as a deepstore and the minion logs are showing this exception: `java.lang.IllegalStateException: PinotFS for scheme: s3 has not been initialized` . Is there a way to configure the minions to be able to read from S3? I couldn't find anything in the docs. Thanks!

did you configure pinot fs inside minion job?

Ah I'm not sure how to set that in a minion job. Is there a doc for that?

Would I set it in the offline table corresponding to the realtime table under "ingestionConfig"?

yes, it should be there with the pinot fs configs

I see. Currently I'm setting all the S3 details in the controller/server config files. Is it recommended to set them in each table instead?

```{
  "tableName": "airlineStats",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "timeColumnName": "DaysSinceEpoch",
    "timeType": "DAYS",
    "segmentPushType": "APPEND",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
    "replication": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP"
  },
  "metadata": {
    "customConfigs": {}
  },
  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "DAILY",
      "batchConfigMaps": [
        {
          "inputDirURI": "<s3://my.s3.bucket/batch/airlineStats/rawdata/>",
          "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
          "input.fs.prop.region": "us-west-2",
          "includeFileNamePattern": "glob:**/*.avro",
          "excludeFileNamePattern": "glob:**/*.tmp",
          "inputFormat": "avro",
          "outputDirURI": "<s3://my.s3.bucket/batch/airlineStats/segments>",
          "push.mode": "metadata"
        }
      ],
      "segmentNameSpec": {},
      "pushSpec": {}
    }
  },
  "task": {
    "taskTypeConfigsMap": {
      "SegmentGenerationAndPushTask": {
      }
    }
  }
}```


controller/servers have it as pinot uses s3 for deep store, however you may have different access credential for ingestion data bucket

you can configure your accesskey and secret along
```"input.fs.prop.region": "us-west-2",```


Got it, makes sense. For configuring in the minion config files, what keys do I set? I tried using something like `pinot.minion.storage.factory.s3.accessKey` but that doesn't seem to work

try configs without `pinot.minion` prefix. cc <@UDT7GFEG6> to confirm

removing the pinot.minion prefix seems to have worked. thanks!

I have another quick question about the RealtimeToOfflineSegmentsTask <@UGRJA9TEH>. When I set the bucketTimePeriod to something like 1d, does Pinot round it to whole time periods? For example, would it strictly separate data such that one segment would only contain data for 2021-10-25. Or does it do a relative time period from whenever the task is run, so a segment could contain data from 2021-10-25 and 2021-10-24 if it was run mid day?

<@U02BHA75K52> It does strict partition based on the epoch time

So each segment only contains data from the same epoch day