https://pinot.apache.org/ logo
#general
Title
# general
p

Pedro Silva

05/04/2021, 10:46 AM
Hello, are Pinot helm charts designed to define sensitive information such as credentials for deep storage in configMaps?
Say I want to configure an S3 bucket for deep storage as specified in: https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot Is there any way to define bucket credentials in Secrets in order to not expose them in clear text through the controller ConfigMap?
d

Daniel Lavoie

05/04/2021, 11:48 AM
As of today, pinot properties are exclusively built from configMap. Merging Pinot Properties between confimap and secrets would be a nice addition to the helm chart. Feel free to contribute a PR if you feel like it. Happy to provide implementation guidelines and reviews.
I’m also trying to motivate the Pinot community to introduce Secret as a first class SPI into Pinot.
1
But that’s a long term effort 🙂
p

Pedro Silva

05/04/2021, 11:53 AM
Thank you for the information. What is SPI?
From what I know of Pinot, simply adding secret support is not enough. Pinot needs to support configuration through environment variables for secrets + configmaps to work if I understand correctly.
d

Daniel Lavoie

05/04/2021, 11:57 AM
All pinot is support for now is properties. Which we can generate on pod provisioning by merging the existing configMap with a k8s secret.
Some specific deep store providers supports Env variable, but that is managed by the underlying client lib, not Pinot. S3 for example.
What is SPI?
It
Service Provider Interface
. Anything that is pluggable in Pinot is managed through predefined interfaces. We call those SPI.
p

Pedro Silva

05/04/2021, 12:03 PM
“All pinot is support for now is properties. Which we can generate on pod provisioning by merging the existing configMap with a k8s secret.” Is there any existing configuration that is generated this way from which I could get some inspiration?
d

Daniel Lavoie

05/04/2021, 12:06 PM
Actually, what I had in mind wouldn’t work. The secrets would endup in the configfile anyway
What we would need is phase 2 of this refactor: https://github.com/apache/incubator-pinot/pull/5608
It would allow loading environment variables as Pinot properties
Hence we could configure secrets are environment variables
p

Pedro Silva

05/04/2021, 12:55 PM
Looks exactly like what I was looking for. Is phase2 of this a work in progress that I can follow?
d

Daniel Lavoie

05/04/2021, 1:44 PM
Unfortunately this is not something I personally planned on working. If you feel like contributing to Pinot, once again, I’ll be happy to provide you pointer for phase 2.
c

Chethan UK

05/05/2021, 8:13 PM
Some specific deep store providers supports Env variable, but that is managed by the underlying client lib, not Pinot. S3 for example
Is mounting cred.json and reading from file supported?
If so, users can mount those files via vault and init containers right
d

Daniel Lavoie

05/05/2021, 8:15 PM
For S3 you can configure everything as secret mounted as env variable for sure. But the helm chart does not support it yet
Same for GCP credentials.json