https://pinot.apache.org/ logo
#general
Title
# general
p

Pavel Stejskal

02/14/2022, 3:05 PM
Hello, has someone success with Minio S3 as a deep storage backend?
k

Kishore G

02/14/2022, 3:07 PM
I am not able to recollect but I remember someone getting it to work.. are you facing any issues?
p

Pavel Stejskal

02/14/2022, 3:09 PM
I’m Im not sure how to test it, I don't see any error but neither any files in bucket when segment is completed
k

Kishore G

02/14/2022, 3:16 PM
Have you configured the fs in controller?
p

Pavel Stejskal

02/14/2022, 3:17 PM
extra: configs: |- pinot.set.instance.id.to.hostname=true controller.task.scheduler.enabled=true controller.enable.split.commit=true controller.data.dir=s3://pinot/ pinot.controller.storage.factory.s3.accessKey=* pinot.controller.storage.factory.s3.secretKey=* pinot.controller.segment.fetcher.protocols=file,http,s3 pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3.endpoint=http://minio-democ.minio-tenant-1:9000 pinot.controller.storage.factory.s3.region=us-west-2 pinot.controller.storage.factory.s3.disableAcl=false
j

Jeff Moszuti

02/14/2022, 3:31 PM
You might need to remove controller.extra.configs.controller.data.dir=s3://pinot and instead define the location of the s3 bucket using controller.data.dir. Refer to this thread for further details. I have Minio S3 working as a deep storage backend for my development environment.
🙏 2
p

Pavel Stejskal

02/14/2022, 3:51 PM
Perfect! It’s working. Controller needs to have dir set to s3:// in helm values and in other config I have controller.local.temp.dir=/tmp/pinot-tmp-data/ …
😎 1
m

Mayank

02/14/2022, 4:15 PM
@User seems like a good one to document.
👍 1
2 Views