Hi Team -- A basic question , If we use S3 for sto...
# generala
Ambika
05/11/2021, 2:58 PMHi Team -- A basic question , If we use S3 for storing the segments, how does pinot take care of query latency since there will be netw call involved?
p
Pedro Silva
05/11/2021, 3:00 PMSomeone with more experience can chime in but Pinot controller & server instances will have a local copy of the segments to handle queries.
Pedro Silva
05/11/2021, 3:01 PMk
Kishore G
05/11/2021, 3:02 PM
Only Pinot server will have the local copy, not controller
a
Ambika
05/11/2021, 3:02 PMok.. so does it mean the cluster must have enough memory combined to host all the data that is getting ingested?
Ambika
05/11/2021, 3:03 PMand that would keep growing in production, so how do we generally scale that ? Should we be looking at sacling up number of servers as traffic and data grows?
k
Kishore G
05/11/2021, 3:03 PM
No, all the data is stored on disk and mmaped. So you need enough disk space
a
Ambika
05/11/2021, 3:03 PMok.. what is we use S3 ?
k
Kishore G
05/11/2021, 3:03 PM
s3 the backup store
a
Ambika
05/11/2021, 3:04 PMoh ok.. so we will have data in disk of the servers +the backup store.
m
Mayank
05/11/2021, 3:06 PM
Correct
p
Pedro Silva
05/11/2021, 3:14 PM@User So when configuring S3 or some other object store as deep store for segments means that disk space in servers is for caching purposes only?
Pedro Silva
05/11/2021, 3:15 PMAre there any metrics/ways to see if see if a given server is under-provisioned for caching segments?
m
Mayank
05/11/2021, 3:15 PM
Yes, servers use local disk for better performance, as of now
Mayank
05/11/2021, 3:16 PM
There’s an endpoint in server to get table size.
✔️ 1