https://pinot.apache.org/ logo
t

Tanmay Movva

10/20/2020, 5:35 PM
Hello, I am trying to setup s3 as segment store for pinot, which is deployed on kubernetes. Unfortunately it is a cross account bucket and we have to pass bucket ACL also. I couldn’t find any way to pass acl policy in the docs. Can anyone please help me with this?
x

Xiang Fu

10/20/2020, 6:07 PM
You can try to set it in controller/server config like:
Copy code
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.storage.factory.s3.accessKey=AKIARC**********
pinot.server.storage.factory.s3.secretKey=aaaaaaaaaaaa
similar in controller:
Copy code
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-west-2
pinot.controller.storage.factory.s3.accessKey=AKIARC**********
pinot.controller.storage.factory.s3.secretKey=aaaaaaaaaaaa
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
t

Tanmay Movva

10/20/2020, 6:29 PM
I did get that part. But we have to provide acl policy for the s3 bucket so that pinot is able to write in that bucket. I am looking for something similar to
druid.storage.disableAcl
in druid. Ref - https://druid.apache.org/docs/latest/development/extensions-core/s3.html It’s implementation can be found here - https://github.com/apache/druid/blob/c3cb0e8b02c641746a5225bd3651e6e441437f19/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java#L283
I have already set the required configs for s3. Thanks for your quick reply @Xiang Fu!
But what I need is to tell pinot to set
bucket-owner-full-control
as the acl.
x

Xiang Fu

10/20/2020, 6:33 PM
let me take a look that
we might have to change this code to setup the s3clientbuilder
@Pradeep what do you think?
x

Xiang Fu

10/20/2020, 6:45 PM
can we try to expose those options transparently?
p

Pradeep

10/20/2020, 6:46 PM
code is missing too
k

Kishore G

10/20/2020, 6:51 PM
I think Xiang is suggesting if there is a way we can pass all the properties from pinot.controller.segment.fetcher.s3.** transparently to the s3clientbuilder, this will solve the problem of having to change the code everytime a new property needs to be set in S3Client
👍 1
p

Pradeep

10/20/2020, 6:53 PM
Seems like this acl property need to be setup for each upload/copy (PutObjectRequest & CopyObjectRequest) of any file on S3, so not sure we can achieve that with just properties without code change.
k

Kishore G

10/20/2020, 6:54 PM
I see
p

Pradeep

10/20/2020, 6:56 PM
Code change should be simple, whereever there is CopyObjectRequest or PutObjectRequest and when the config is turned on, set acls https://sourcegraph.com/github.com/apache/druid@c3cb0e8/-/blob/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentMover.java#L196
x

Xiang Fu

10/20/2020, 7:09 PM
got it. Created an issue: https://github.com/apache/incubator-pinot/issues/6161 , Could you fill up more info there ?
p

Pradeep

10/20/2020, 7:14 PM
done
2 Views