Hello I am trying to setup s3 as segment store for pinot whi

Question

Hello I am trying to setup s3 as segment store for pinot which is deployed on kubernetes Unfortunately it is a cross account bucket and we have to pass bucket ACL also I couldn t find any way to pass acl policy in the docs Can anyone please help me with this

Xiang Fu · Answer

You can try to set it in controller server config like ```pinot server storage factory class s3=org apache pinot plugin filesystem S3PinotFS pinot server storage factory s3 region=us west 2 pinot server storage factory s3 accessKey=AKIARC pinot server storage factory s3 secretKey=aaaaaaaaaaaa```

Xiang Fu · Answer

similar in controller ```pinot controller storage factory class s3=org apache pinot plugin filesystem S3PinotFS pinot controller storage factory s3 region=us west 2 pinot controller storage factory s3 accessKey=AKIARC pinot controller storage factory s3 secretKey=aaaaaaaaaaaa pinot controller segment fetcher protocols=file http s3 pinot controller segment fetcher s3 class=org apache pinot common utils fetcher PinotFSSegmentFetcher```

Tanmay Movva · Answer

I did get that part But we have to provide acl policy for the s3 bucket so that pinot is able to write in that bucket I am looking for something similar to `druid storage disableAcl` in druid Ref It s implementation can be found here

Tanmay Movva · Answer

I have already set the required configs for s3 Thanks for your quick reply < Xiang Fu>

Tanmay Movva · Answer

But what I need is to tell pinot to set `bucket owner full control` as the acl

Xiang Fu · Answer

let me take a look that

Kishore G · Answer

Kishore G · Answer

we might have to change this code to setup the s3clientbuilder

Kishore G · Answer

< Pradeep> what do you think

Pradeep · Answer

< Tanmay Movva> I think S3PinotFS needs to be updated it seems currently that option is missing something similar to what you pointed out here should work I believe for all the PutObjectRequests

Xiang Fu · Answer

can we try to expose those options transparently

Pradeep · Answer

code is missing too

Kishore G · Answer

I think Xiang is suggesting if there is a way we can pass all the properties from pinot controller segment fetcher s3 transparently to the s3clientbuilder this will solve the problem of having to change the code everytime a new property needs to be set in S3Client

Pradeep · Answer

Seems like this acl property need to be setup for each upload copy PutObjectRequest amp CopyObjectRequest of any file on S3 so not sure we can achieve that with just properties without code change

Kishore G · Answer

I see

Pradeep · Answer

Code change should be simple whereever there is CopyObjectRequest or PutObjectRequest and when the config is turned on set acls <https sourcegraph com github com apache druid c3cb0e8 blob extensions core s3 extensions src main java org apache druid storage s3 S3DataSegmentMover java L196>

Xiang Fu · Answer

got it Created an issue <https github com apache incubator pinot issues 6161> Could you fill up more info there

Pradeep · Answer

done