Apache Pinot

I am ingesting a table from a Kafka topic with
```        "realtime.segment.flush.threshold.rows": "0",
        "realtime.segment.flush.threshold.time": "4h",
        "realtime.segment.flush.threshold.segment.size": "40M",```
and I am storing segment files in S3. I am seeing a huge number of files in S3 like
```TABLE__0__0__20211029T1835Z.tmp.420158c9-1742-4bd2-bbae-5a59d2205cd2```
that are all much smaller than 40M. There are several thousand files. What are these?

40M seems on the smaller side, what’s your real-time server size (cpu/mem)?

Also did guy configure temp dir to be in s3?

This is a staging environment and the data volume is relatively low, thus the 40M. In production it will be 500M.
And the temp dir is not in S3
```      controller.data.dir=<s3://BUCKET_NAME/segments>
      controller.local.temp.dir=/var/pinot/controller/data```

After 3 days the S3 bucket has 1.1M files and 520GB of data - but I think I might have found the issue. Will update...

After getting the server and controller configuration in sync and restarting, I am still seeing huge numbers of files, mostly of the form `TABLE5Z.tmp.0049f61a-96b0-4cc0-997c-07baf816a92b`

Seeing this in the server logs
```2021/11/02 15:38:06.103 INFO [ServerSegmentCompletionProtocolHandler] [network_event_changes__9__0__20211101T2025Z] Controller response {"offset":-1,"streamPartitionMsgOffset":null,"status":"FAILED","isSplitCommitType":false,"buildTimeSec":-1} for <http://pinot-controller-2.pinot-controller-headless.pinot.svc.cluster.local:9000/segmentCommitEndWithMetadata?segmentSizeBytes=1811784&amp;buildTimeMillis=254&amp;streamPartitionMsgOffset=18234&amp;instance=Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098&amp;offset=-1&amp;name=network_event_changes__9__0__20211101T2025Z&amp;location=s3%3A%2F%2Fpinot-sdar0xqh11ly8dclm%2Fsegments%2Fnetwork_event_changes%2Fnetwork_event_changes__9__0__20211101T2025Z.tmp.59506639-5dce-42d7-973b-4d0f389cd28a&amp;rowCount=1204&amp;memoryUsedBytes=45839852>
2021/11/02 15:38:06.103 WARN [LLRealtimeSegmentDataManager_network_event_changes__15__0__20211101T2025Z] [network_event_changes__9__0__20211101T2025Z] CommitEnd failed with response {"offset":-1,"streamPartitionMsgOffset":null,"status":"FAILED","isSplitCommitType":false,"buildTimeSec":-1}
2021/11/02 15:38:06.103 INFO [LLRealtimeSegmentDataManager_network_event_changes__9__0__20211101T2025Z] [network_event_changes__9__0__20211101T2025Z] Could not commit segment. Retrying after hold```

I found it -- I had insufficient S3 permissions (the S3 permissions required for Pinot do not appear to be documented). Working now

For anybody that might find this in the future, the S3 permissions that worked for me are
```      "s3:DeleteObject",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:PutObject",
      "s3:HeadObject"```