hi Team, can someone provide me some context on m...
# troubleshooting
p
hi Team, can someone provide me some context on my query below related to how s3 is involved/behaves when used as a deepstore. my current setup. pinot configure to bypass controller and using se as deepstore so server push and pull data directly from deepstore. i believe their is a config that can be defined (age i guess) in realtime table so realtime segments are still kept on disk. but we are not using this. recent problem faced. recently we had outage and lost connectivity with s3 from pinot which resulted in ingestion being stopped. this seems our mistake as along with bypassing controller we need to configure realtime table to keep ingesting and keeping segments locally while s3 gets back up. my queries. also currently we have intermittent http timeouts with s3. due to this segments external state goes to ERROR and table goes to bad state. while we are working on the network side of the issue i need to understand below. 1> what is the involvement of s3 is it just a deepstore for backup. even with our config does pinot still keep segments on disk. 2> when query is executed does it always go to s3 or still queries data on disk provided pinot keeps it and only for segment not on disk query goes to s3. i am trying to understand how and when s3 is queried. is it used for upload, query or both. below is the error we face currently i need to understand the behavior to setup pinot better. if any document is their related same it will help Caught exception while fetching segment from: s3://xyz/abc/abc__31__73__20250604T0211Z to: /var/pinot/server/data/index/abc_REALTIME/tmp/tmp-abc__31__73__20250604T0211Z-7b64a679-f65a-4325-abc8-956b372fea55/abc__31__73__20250604T0211Z.tar.gz org.apache.pinot.shaded.software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connect to xyz:9020 [xyz/0.0.0.0]
m
1. Deepstore is for data backup / durability. Servers do keep a local copy for faster processing. 2. Deepstore is not in query path, query is answered from servers local copy. When realtime segment is committed, or offline segment is pushed, these events will write to deepstore S3. When server is missing the local copy (new data or lost copy or server) it reads from deepstore.
p
Thanks for the reply. 1 supplementary question. what config decides how much data is stored on disk while i believe all is stored in S3. (i am assuming what is stored in disk will result in pinot to get segment from s3 if it does not find this segment on disk as u mentioned)
m
In OSS, all data is also stored on local disk of servers (each server hosts part of it).
p
ok so all data is saved on both places disk and s3. so s3 just acts as a backup or mechanism to recover from disaster. Thanks for clarifying correct me if my assumption is wrong.
s
So deepstore , it's just for backups? I was wondering if there is a way we could have moved the data to deepstore after retension period in the table has reached and stays in deepstore untill deepstore retension seperate from realtime table tension has reached. I don't see a way for this as of now so that we can treat the hdfs as archive + disaster recovery. The current behaviour is completed segments gets stored both on server aswell as deepstore Untill retension period is reached for realtime table and then I'm both controller and deepstore there will be a deleted segments folder created which will be retained 7 days by default. Let's say 30 days is retension on realtime table On 31st day once retension manager ecans that it has reached its end of life 30 days old data gets deleted in both places ! @Mayank I need this clarification to consider deepstore as our archive + backup @Mayank @Subbu Subramaniam is my understanding correct? #troubleshooting
m
deletedSegmentsRetentionPeriod
set this to 0, then data will be deleted from both places.
s
I want to store hdfs data longer than table retension period. @Mayank
m
For what purpose? You cannot query that data using Pinot
s
Archival is the goal
m
Again, for what purpose? This data is in Pinot segment format. Other systems cannot read it, Pinot won't read it until it is part of Pinot cluster (not deletedSegments)
s
Can't trino read from archival place let's say hdfs? If I configure trino - pinot connector seperately? ( It has its own broker limit issues though )
m
That
deletedSegments
folder is for deleted data, and for recovery. And the only way to recover is to push the data back into Pinot manually. Unless you do that, Pinot can't read it, Trino can't read it, no other system will be able to read it
s
Got it. So seperate automation is the only way we need to take backup of the data if we want to retain longer. And it won't be queryable
Thanks
@Mayank once retension is reached , Can you confirm Whether segments gets deleted from both server as well as deestorage ?
m
It is. From deepstore it might go to deletedSegments folkder if that config is set for the configured time