Hi guys, I tried setting up S3 deep store for my Pinot cluster, and used this part of config in the

pinot.yaml

file, in the

controller

config, for deploying via the official Helm chart:

extra:
    # Note: Extra configs will be appended to pinot-controller.conf file
    configs: |-
      pinot.set.instance.id.to.hostname=true
      controller.task.scheduler.enabled=true
      # Note: change this to the real bucket, after creating it in S3
      controller.data.dir=s3://<redacted>
      controller.local.temp.dir=/tmp/pinot-tmp-data/
      pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      pinot.controller.storage.factory.s3.region=<redacted>
      pinot.controller.segment.fetcher.protocols=file,http,s3
      pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

Is this the correct configuration to do - to append these configs to the existing ones? Or should the approach have been different? I'm asking this because I found a weird directory in one of the Controller local filesystems:

root@pinot-controller-1:/opt/pinot# du -sh /var/pinot/controller/data\,s3\:/
0	/var/pinot/controller/data,s3:/

By the looks of it, it seems like a configuration mistake somewhere [SOLVED! - SOLUTION:] Because the official Helm chart already defines that configuration, it ends up concatenating the one from the extras and the default one when loading from the final file, because that setting ends up defined twice in

pinot-controller.conf

inside the Controller container. The solution for this is to, instead of putting

controller.data.dir

as a one-line string in the

extra

configs

, just define that setting starting from the

controller

options in that YAML file, then

data

instead of

extra

, then

dir

, so that the option replaces the default value.

According to the following document, the plugin

pinot-s3

needs to be added to the JVM args of controller (and minion)

-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-s3

From our logs, though, that plugin seems to have been loaded already

Ok, then I guess this JVM arg is no longer needed. That was the only difference I could see between my configuration. Otherwise, my config map is the same as yours. I also appended the S3 details to the existing one.

Ah, cool! Thanks man! 🙂

re: adding the plugin explicitly - I think you only need to do that if you don't want ALL plugins to be loaded. re: The

controller.data.dir

- you are right that it seems to be appending that value to another value. I'm not sure how we can get it to override the initial value. I guess @Mayank might know

Weird... well, I'm keeping an eye on things...

the problem is that it may be writing the segment backup files to that weird concatenated path! Keep an eye if it's doing that. If you see the segment files in S3 then it's probably fine.

Correct, and it's not fine, it's putting the files there indeed:

root@pinot-controller-0:/opt/pinot# du -sh /var/pinot/controller/data,s3:/
166M	/var/pinot/controller/data,s3:/

and I see no objects in my S3 bucket

hmmm @Ahmed Shehata had pretty much the same problem on this thread https://apache-pinot.slack.com/archives/C011C9JHN7R/p1638982572389100?thread_ts=1638959396.366700&cid=C011C9JHN7R

Ah, thanks man! Lemme try to find that out...

I suppose we should improve user experience by avoiding/flagging such side effects.

If that's possible to do, I think it would be a good idea... or perhaps improving the docs, maybe, I dunno...

winning! Could you share the working config and where in the docs needs updating and then I'll update the docs tomorrow?

Thanks @Mark Needham @Diogo Baeder

Sure! By the way, it would be nice to have some docs about configuring all this together, perhaps - deploying Pinot in Amazon EKS with the Helm chart and S3 as deep store. I think it might be a common thing for other projects, maybe.

yeh that's a good idea

Yeah, that would be great! Maybe some recipes, I dunno...

perfect. Thanks very much!

❤️

just to check. That's your values.yaml file right? Presumably based on https://github.com/apache/pinot/blob/82fe4fae32bdba37cdaa80a0874d60453d462c27/kubernetes/helm/pinot/values.yaml

That's correct, although it's not clear to me exactly how they're merged with the other values files (I'm not even sure if we use the official values files, perhaps we don't, but even so these configs I sent I believe are relevant - but might require some reviewing from your guys, of course)