Hello, I am trying to setup s3 as segment store fo...
# troubleshootingt
Tanmay Movva
10/20/2020, 5:35 PMHello, I am trying to setup s3 as segment store for pinot, which is deployed on kubernetes. Unfortunately it is a cross account bucket and we have to pass bucket ACL also. I couldn’t find any way to pass acl policy in the docs. Can anyone please help me with this?
x
Xiang Fu
10/20/2020, 6:07 PM
You can try to set it in controller/server config like:
Copy code
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.storage.factory.s3.accessKey=AKIARC**********
pinot.server.storage.factory.s3.secretKey=aaaaaaaaaaaaXiang Fu
10/20/2020, 6:27 PM
similar in controller:
Copy code
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-west-2
pinot.controller.storage.factory.s3.accessKey=AKIARC**********
pinot.controller.storage.factory.s3.secretKey=aaaaaaaaaaaa
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetchert
Tanmay Movva
10/20/2020, 6:29 PMI did get that part. But we have to provide acl policy for the s3 bucket so that pinot is able to write in that bucket.
I am looking for something similar to
druid.storage.disableAclTanmay Movva
10/20/2020, 6:29 PMI have already set the required configs for s3. Thanks for your quick reply @Xiang Fu!
Tanmay Movva
10/20/2020, 6:30 PMBut what I need is to tell pinot to set
bucket-owner-full-controlx
Xiang Fu
10/20/2020, 6:33 PM
let me take a look that
Kishore G
10/20/2020, 6:39 PM
we might have to change this code to setup the s3clientbuilder
Kishore G
10/20/2020, 6:39 PM
@Pradeep what do you think?
p
Pradeep
10/20/2020, 6:45 PM@Tanmay Movva I think S3PinotFS needs to be updated it seems, currently that option is missing
https://sourcegraph.com/github.com/apache/incubator-pinot/-/blob/pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java#L79:11
something similar to what you pointed out here should work I believe for all the PutObjectRequests
https://github.com/apache/druid/blob/c3cb0e8b02c641746a5225bd3651e6e441437f19/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java#L283
x
Xiang Fu
10/20/2020, 6:45 PM
can we try to expose those options transparently?
p
Pradeep
10/20/2020, 6:46 PMcode is missing too
k
Kishore G
10/20/2020, 6:51 PM
I think Xiang is suggesting if there is a way we can pass all the properties from pinot.controller.segment.fetcher.s3.** transparently to the s3clientbuilder, this will solve the problem of having to change the code everytime a new property needs to be set in S3Client
👍 1
p
Pradeep
10/20/2020, 6:53 PMSeems like this acl property need to be setup for each upload/copy (PutObjectRequest & CopyObjectRequest) of any file on S3, so not sure we can achieve that with just properties without code change.
k
Kishore G
10/20/2020, 6:54 PM
I see
p
Pradeep
10/20/2020, 6:56 PMCode change should be simple, whereever there is CopyObjectRequest or PutObjectRequest and when the config is turned on, set acls
https://sourcegraph.com/github.com/apache/druid@c3cb0e8/-/blob/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentMover.java#L196
x
Xiang Fu
10/20/2020, 7:09 PM
got it. Created an issue: https://github.com/apache/incubator-pinot/issues/6161 , Could you fill up more info there ?
p
Pradeep
10/20/2020, 7:14 PMdone
2 Views