So I have been looking into kubernetes lately and ...
# troubleshooting
a
So I have been looking into kubernetes lately and have figured out few things Just wanted to understand what are the issues one face or faced using pinot in AWS EKS . Any challanges that we face during the ingestion or scaling or creating segements , Would love to know . Also is it a good idea to use open source pinot in AWS EKS for production ? Is it running properly without any bugs and also any newer version that has fixed some bugs which were there earlier ?
Also why do we need deep storage (s3) here . I thought we were using EBS for the data mount then what is the reason ?
x
s3 is deep store for data backup and recovery
a
so do I need to regularly push my data in s3 for back ups? I am guessing EBS will take care of it tho ? How often do you suggest back ups to deep storage should be done
x
no, for all the segments pushed to pinot cluster, pinot auto persist those into s3
a
ok ok . so I need to configure that into my values.yaml when to persist them to s3 ? like will it be mentioned in the .yaml file ?
x
you just need to configure s3 as the deep store in pinot controller and servers
then all the segments will be copied to s3 for backup automatically
a
Hi @Xiang Fu we will be initating a Pinot cluster on AWS EKS . I will keep you updated on the same if any issue but can you give some suggestions how many nodes a cluster I should keep, keeping in mind 4-5 million json payloads being inserted in my pinot and each Json has around 100 fileds . some of them are nested as well. Also what type of volumes I should use for an optimal performance
x
what’s the size of each json payload? and your query pattern as well the qps?
i would suggest to build json index for your data and you can check the table size on the UI for the storage.
increase cpu/nodes will help query perf
a
so instead of flattening the data you are telling me to ingest the json as a whole , Wont creating json index make the size big of the data ? I know its faster but I have done a POC where creating index increased the size of the data 3 time the normal . but we have nested json. Ill tell you the size in a while.
x
then you can flat one level then add index based on your needs
a
got it