Hello, congratz on the tiered storage!
I'm readin...
# generalj
Jaromir Hamala
03/02/2022, 8:10 AMHello, congratz on the tiered storage!
I'm reading the announcement and it says: Note that this is not implemented as lazy-loading - Pinot servers directly query data on the cloud and are never downloading the entire segments locally.
May I ask how does it work? I know close to nothing about S3, but I believe it's a dummy blob-store. You have to download blobs with segments before querying them, don't you? Am I missing anything? Thanks for any hint!
m
Mark Needham
03/02/2022, 11:04 AM
cc @User @User
👍 1
k
Kishore G
03/02/2022, 2:33 PM
We will publish another blog on the details but at a high level S3 has api where you can read a part of the blob and we leverage that
l
Luis Fernandez
03/02/2022, 2:43 PMWhere do we get this announcements?
k
Kishore G
03/02/2022, 2:50 PM
This is on StarTree blog https://www.startree.ai/blogs/introducing-tiered-storage
🙌 1
l
Luis Fernandez
03/02/2022, 2:53 PM😮
Luis Fernandez
03/02/2022, 2:54 PMthis will definitely be beneficial for our use case thank you!
🍷 5
j
Jaromir Hamala
03/02/2022, 3:16 PMIf I understand it right Apache Pinot won't copy the whole blob, but just some parts of it. Thank you!
k
Kishore G
03/02/2022, 3:52 PM
thats right, only the parts that are needed to process the query.
👍 1