1.. Has anyone hooked up Pinot to read files from MinIO (open source S3 alternative)? I'm using MinIO in a local Kubernetes setup (and S3 in prod). I checked out S3PinotFS but I don't see an easy way to override the endpoint for S3. Can this be done using the URI? E.g.

<s3://minio:9000/mybucket/objectpath>

or

<s3://mybucket.minio:9000/objectpath>

? 2.. What's the best way to add the

pinot-s3

plugin with

apachepinot/pinot

docker image? Do I need to create my own wrapping image? I see a few environment variables in

pinot-admin.sh

that I can set. E.g.

JAVAOPTS

,

PLUGINS_INCLUDE

,

PLUGINS_CLASSPATH

,

PLUGINS_DIR

see if this doc can solve your problem:

Currently, endpoint override is not supported

1.. Cool, I found the other s3 page. I'll try the instructions in this one.

Ohh if that's the case then it should be pretty easy to make the code change. As long as endpoint is not part of the path

Hi I stumbled on this thread and was wondering whether this was ever implemented ? I am performing a PoC on using Apache Pinot for a specific use case. Being able to work with minIO is necessary, hence for deepstorage but also ingesting from it. Any updates on this ? I saw some examples for executionFrameworkSpec, but I am looking for batchIngestionConfig

@Yoav Nordmann Tutorial and configs

Thanks @adheip singh I was actually able to do this the following way: I added

input.fs.prop.endpoint

to the

batchConfigMaps

and added the same value as given in

pinot.controller.storage.factory.s3.endpoint