hi Team,
can someone provide me some context on my query below related to how s3 is involved/behaves when used as a deepstore.
my current setup.
pinot configure to bypass controller and using se as deepstore so server push and pull data directly from deepstore.
i believe their is a config that can be defined (age i guess) in realtime table so realtime segments are still kept on disk. but we are not using this.
recent problem faced.
recently we had outage and lost connectivity with s3 from pinot which resulted in ingestion being stopped. this seems our mistake as along with bypassing controller we need to configure realtime table to keep ingesting and keeping segments locally while s3 gets back up.
my queries.
also currently we have intermittent http timeouts with s3. due to this segments external state goes to ERROR and table goes to bad state. while we are working on the network side of the issue i need to understand below.
1> what is the involvement of s3 is it just a deepstore for backup. even with our config does pinot still keep segments on disk.
2> when query is executed does it always go to s3 or still queries data on disk provided pinot keeps it and only for segment not on disk query goes to s3.
i am trying to understand how and when s3 is queried. is it used for upload, query or both. below is the error we face currently i need to understand the behavior to setup pinot better. if any document is their related same it will help
Caught exception while fetching segment from:
s3://xyz/abc/abc__31__73__20250604T0211Z to: /var/pinot/server/data/index/abc_REALTIME/tmp/tmp-abc__31__73__20250604T0211Z-7b64a679-f65a-4325-abc8-956b372fea55/abc__31__73__20250604T0211Z.tar.gz org.apache.pinot.shaded.software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connect to xyz:9020 [xyz/0.0.0.0]