Hi we want to use google cloud storage as a pluggable storag Apache Pinot #general

Hi, we want to use google cloud storage as a plugg...

Elon

11/27/2019, 12:38 AM

Hi, we want to use google cloud storage as a pluggable storage, does anyone have any example configs or advice on how to do that? cc: @User @User

Xiang Fu

11/27/2019, 12:41 AM

Do you mean use gs buckets?

Elon

11/27/2019, 12:42 AM

Hi, yes, that's what I mean:)

Xiang Fu

11/27/2019, 12:42 AM

I suggest to use persistent disk for Pinot server

Xiang Fu

11/27/2019, 12:43 AM

gs could be used as controller deep store

Elon

11/27/2019, 12:43 AM

For the temp directory or are you saying we shouldn't use gcs buckets at all?

Elon

11/27/2019, 12:43 AM

Ah, makes sense - so how would we use gcs for controller deep store?

Xiang Fu

11/27/2019, 12:44 AM

The thing I’ve tried out is to make a nfs on top of gcs

Elon

11/27/2019, 12:44 AM

Nice! How do you do that? Or is there a pull request or code you can point me to?

Xiang Fu

11/27/2019, 12:44 AM

Then mount the nfs to all controller pods

Xiang Fu

11/27/2019, 12:44 AM

There is an open source repo of gcsfuse

Elon

11/27/2019, 12:45 AM

A cool, I just got it

Elon

11/27/2019, 12:46 AM

So you would use the gcsfuse utility in the helm chart (init job?) to mount the bucket to the path?

Elon

11/27/2019, 12:48 AM

Ah I see - you add it to the image and then run gcsfuse as part of the start up?

Elon

11/27/2019, 12:49 AM

Thanks for the advice!

Xiang Fu

11/27/2019, 12:50 AM

https://github.com/maciekrb/gcs-fuse-sample

Xiang Fu

11/27/2019, 12:52 AM

https://medium.com/grensesnittet/mounting-a-gcp-bucket-as-nfs-in-kubernetes-8f6d3faf4da3

Elon

11/27/2019, 12:52 AM

Nice, I'll try this out. This is better than using the HadoopPinotFS with gcs?

Xiang Fu

11/27/2019, 12:53 AM

There are two ways, if you can build your own image, then you can have one pod which mount gcs during pod init

Elon

11/27/2019, 12:54 AM

Ok, and that's for the controller pods, right?

Xiang Fu

11/27/2019, 12:54 AM

The other is that you maintain a nfs service and deployment which mount gcs then your Pinot controller pod just need to mount a nfs

Elon

11/27/2019, 12:54 AM

Ah ok

Elon

11/27/2019, 12:54 AM

I'm not sure I know how to do that one, the link you shared describes how to do that?

Elon

11/27/2019, 12:54 AM

The second option I mean

Xiang Fu

11/27/2019, 12:55 AM

For server we suggest to mount persistence disk

Xiang Fu

11/27/2019, 12:55 AM

Yes

👍 1

Xiang Fu

11/27/2019, 12:55 AM

https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes this is the disk for Pinot servers

👍 1

Xiang Fu

11/27/2019, 12:56 AM

In helm, you can change storageclass for Pinot server to get this

Elon

11/27/2019, 12:57 AM

Nice. We have pvc's working right now - but for the controller to use the gcsfuse mount what config parameters would I need to set?

Xiang Fu

11/27/2019, 12:57 AM

Also note that this is per server per gce disk

Xiang Fu

11/27/2019, 12:57 AM

Yes

Elon

11/27/2019, 12:57 AM

Yep

Xiang Fu

11/27/2019, 12:57 AM

You need the change the volume

Elon

11/27/2019, 12:58 AM

Is that the

Copy code

controller.data.dir

Elon

11/27/2019, 12:58 AM

parameter?

Xiang Fu

11/27/2019, 1:00 AM

Yes

Xiang Fu

11/27/2019, 1:01 AM

This should be sub directory under your gcs mount

👍 1

Elon

11/27/2019, 1:04 AM

Thanks a lot! Sounds like this should be enough to get started.

Elon

11/27/2019, 1:05 AM

And just one more n00b question (still new to pinot) - what's the advantage for using gcs for controller deep storage?

Xiang Fu

11/27/2019, 1:22 AM

Cause for pint controller to serve segments, you want a volume mounted to all the pods, which is ReadWriteMany

Elon

11/27/2019, 1:22 AM

Got it, thanks

Xiang Fu

11/27/2019, 1:22 AM

This access mode on gcp is only available as nfs

👍 1

Elon

11/27/2019, 1:24 AM

Nice, thanks for the info!

Open in Slack

Previous Next