Hello, are Pinot helm charts designed to define se...
# generalp
Pedro Silva
05/04/2021, 10:46 AMHello, are Pinot helm charts designed to define sensitive information such as credentials for deep storage in configMaps?
Pedro Silva
05/04/2021, 10:47 AMSay I want to configure an S3 bucket for deep storage as specified in: https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot
Is there any way to define bucket credentials in Secrets in order to not expose them in clear text through the controller ConfigMap?
d
Daniel Lavoie
05/04/2021, 11:48 AMAs of today, pinot properties are exclusively built from configMap. Merging Pinot Properties between confimap and secrets would be a nice addition to the helm chart. Feel free to contribute a PR if you feel like it. Happy to provide implementation guidelines and reviews.
Daniel Lavoie
05/04/2021, 11:48 AMI’m also trying to motivate the Pinot community to introduce Secret as a first class SPI into Pinot.
➕ 1
Daniel Lavoie
05/04/2021, 11:48 AMBut that’s a long term effort 🙂
p
Pedro Silva
05/04/2021, 11:53 AMThank you for the information. What is SPI?
Pedro Silva
05/04/2021, 11:54 AMFrom what I know of Pinot, simply adding secret support is not enough. Pinot needs to support configuration through environment variables for secrets + configmaps to work if I understand correctly.
d
Daniel Lavoie
05/04/2021, 11:57 AMAll pinot is support for now is properties. Which we can generate on pod provisioning by merging the existing configMap with a k8s secret.
Daniel Lavoie
05/04/2021, 11:58 AMSome specific deep store providers supports Env variable, but that is managed by the underlying client lib, not Pinot. S3 for example.
Daniel Lavoie
05/04/2021, 12:01 PMWhat is SPI?It
Service Provider Interfacep
Pedro Silva
05/04/2021, 12:03 PM“All pinot is support for now is properties. Which we can generate on pod provisioning by merging the existing configMap with a k8s secret.”
Is there any existing configuration that is generated this way from which I could get some inspiration?
d
Daniel Lavoie
05/04/2021, 12:06 PMActually, what I had in mind wouldn’t work. The secrets would endup in the configfile anyway
Daniel Lavoie
05/04/2021, 12:07 PMWhat we would need is phase 2 of this refactor: https://github.com/apache/incubator-pinot/pull/5608
Daniel Lavoie
05/04/2021, 12:08 PMIt would allow loading environment variables as Pinot properties
Daniel Lavoie
05/04/2021, 12:08 PMHence we could configure secrets are environment variables
p
Pedro Silva
05/04/2021, 12:55 PMLooks exactly like what I was looking for. Is phase2 of this a work in progress that I can follow?
d
Daniel Lavoie
05/04/2021, 1:44 PMUnfortunately this is not something I personally planned on working. If you feel like contributing to Pinot, once again, I’ll be happy to provide you pointer for phase 2.
c
Chethan UK
05/05/2021, 8:13 PMSome specific deep store providers supports Env variable, but that is managed by the underlying client lib, not Pinot. S3 for exampleIs mounting cred.json and reading from file supported?
Chethan UK
05/05/2021, 8:13 PMIf so, users can mount those files via vault and init containers right
d
Daniel Lavoie
05/05/2021, 8:15 PMFor S3 you can configure everything as secret mounted as env variable for sure. But the helm chart does not support it yet
Daniel Lavoie
05/05/2021, 8:15 PMSame for GCP credentials.json