Apache Pinot

check if the conf is correct, or if there's an issue with the code

```controller.helix.cluster.name=pinot
controller.host=pinot-controller-0
controller.port=9000
controller.vip.host=pinot-controller-0
controller.vip.port=9000
controller.data.dir=data/
controller.zk.str=pinot-zookeeper:2181
pinot.set.instance.id.to.hostname=true
pinot.controller.storage.factory.class.gcs=org.apache.pinot.filesystem.GcsPinotFS
pinot.controller.storage.factory.gcs.projectId=myProject
pinot.controller.storage.factory.gcs.bucketName=myBucket```

<@UDQU92KBK> does it look right? controller.data.dir=data/ or should it be <gcs://data>

cool, should it be fully qualified -&gt; with a bucket name?

We should fix the doc or make it more intuitive 

I think <@UPK7Y0X0A> wrote tests against new FS, and it was taking relative paths (wasn’t working with fully qualified). I assume something upstream removes bucket name?

The code snippet that I pasted above seems to get the scheme based on datadir 

If you find the issue let’s clean it up

Looks like I need to add a scheme (i.e. "gs") and have different code paths for local files and "gs" scheme

Got it working, will update pull request tomorrow

So I was able to get a local tar file stored on the servers and I see that it created the temp schema directory on the controller (gcs bucket). Is there an endpoint where I can upload a file on gcs to the servers?

Yes. <@UDWEW8T0S> do you have a pointer to the docs for URI based upload?

<@UPK7Y0X0A> <https://pinot.readthedocs.io/en/latest/segment_fetcher.html#push-hdfs-segment-to-pinot-controller> this wiki may help; let me know if you have any questions

Have a quick question - is there a way to specify a "pinot fs" scheme with the /segments endpoint? i.e. if we have segments stored in gcs is there an endpoint that we can post to with the remote url? We are not using the hadoopfs, just going straight to gcs.

then yes, it will fetch to local based on the uri and then do metadata operations

the segment fetcher is hooked up to pinot fs and can support all of them

just need to add the correct configs in the server/controller such as:

<https://pinot.readthedocs.io/en/latest/pluggable_storage.html>