Are their any guidelines / estimates to resource r...
# generalp
Playsted
12/09/2020, 9:37 PMAre their any guidelines / estimates to resource requirements for Pinot, particularly memory / local node storage needed? Eg. for 10 Tb table in deep storage need X Tb locally at nodes and Y memory for reasonable performance? Is it designed so that the entire table should be cached locally and MMAP'd or can parts go cold and be pulled on demand from deep storage with the extra latency?
x
Xiang Fu
12/09/2020, 9:44 PM
usually we recommend r5.2xlarge(8 vcore, 64gb ram) + 2TB SSD EBS, you can also use r5.4xlarge + 4TB
Xiang Fu
12/09/2020, 9:45 PM
current pinot requires all data to be on local disk for mmap
p
Playsted
12/09/2020, 9:45 PMDeep storage is really just DR then?
x
Xiang Fu
12/09/2020, 9:46 PM
deep storage right now is not on query serving path
👍 1
Xiang Fu
12/09/2020, 9:46 PM
there is an effort to make the lazy loading for data: https://github.com/apache/incubator-pinot/pull/6250
Xiang Fu
12/09/2020, 9:47 PM
it’s still ongoing
p
Playsted
12/09/2020, 9:48 PMOkay great, that's what I was looking for. Just experimenting now but will assume need local SSD storage equivalent to data set size for now
Playsted
12/09/2020, 9:48 PMThank you
x
Xiang Fu
12/09/2020, 9:49 PM
right, pinot data segment size typically is smaller than avro (50%-60%) just fyi
Xiang Fu
12/09/2020, 9:49 PM
almost same as parquet
p
Playsted
12/09/2020, 9:50 PMGreat, have snappy compressed parquet currently so easy to compare then
g
Guillaume Loetscher
12/10/2020, 8:08 AMusually we recommend r5.2xlarge(8 vcore, 64gb ram) + 2TB SSD EBS, you can also use r5.4xlarge + 4TBInteresting. When you say you are using such server types, is it only for offline / realtime servers ? Or are you running all pinot components on those server types (as you are running docker containers) ?
Guillaume Loetscher
12/10/2020, 8:12 AMI will definitely do some testing and sizing with live data, but it would be nice to have some guidelines in the documentation. Something like :
• default minimal requirements for pinot server : x vCPU, y GB RAM
• additional cpu / RAM per thousand of queries on the broker
• additional cpu / RAM per segments on offline server
• etc…
Guillaume Loetscher
12/10/2020, 8:13 AMit may vary widely between use cases, but at least, it gives some rough estimation on where you should start
x
Xiang Fu
12/10/2020, 9:23 AM
it’s for pinot-server. For broker and controllers, we use m5.xlarge/m5.2xlarge depends on the use case
g
Guillaume Loetscher
12/10/2020, 9:23 AMok
Guillaume Loetscher
12/10/2020, 9:24 AMjust for my understanding, you said you are running Docker in production, but you are using r5.2xlarge for pinot servers.
x
Xiang Fu
12/10/2020, 9:24 AM
k8s
g
Guillaume Loetscher
12/10/2020, 9:24 AMok
Guillaume Loetscher
12/10/2020, 9:26 AMso that’s a kubernetes node and you dedicate it to run pinot servers on it ?
Guillaume Loetscher
12/10/2020, 9:30 AM(also, it’s really late for you, we can continue that conversation tomorrow morning)
x
Xiang Fu
12/10/2020, 9:35 AM
Yes, config nodegroup for machine sku and tag them then we can assign the corresponding pods to them