Team , Im testing data to be pulled from deepstora...
# troubleshooting
m
Team , Im testing data to be pulled from deepstorage in the event of pinot server goes down. Here is the test env: Pinot Cluster : 1.2 Table : airlineStats Total Rows : 276 Deep Storage : HDFS HDFS data : Available (hdfs://pinotperf/pinot/airlineStats/airlineStats_16082_16082_0) Pinot Segement: Resides in Pinot server008 replication : "1" Now if I shutdown server008. Since data lies only in 008 , cluster should try to fetch data from deep storage automatically is my understanding. But that doesn't happen. I see the status of the segment says updating, but no data when we try to query. Is this is the way it should work?
v
That's because the RF=1 that's why you see them updating. There's no other healthy replica available for the segment. Also for fetching the data from deepstore, I don't think it happens automatically until you throw a query that looks for historical data that has been put in hdfs or s3.
x
if that server is down then the data is not served. There is no auto rebalance for pinot.
m
So what is the manual way to make it fetch from deep store?
x
get the server up or rebalance the table
in this case, you should have more replicas for the data serving
so one server down won’t impact your data availability
m
Lets assume server has gone down and not coming up at all.
Agreed, If I have kept 2 replica , the chance is possible. But in a 100node cluster, 2 pinot nodes going down is possible and it might go for complete replacement of 2servers that will come back without data. This way data is gone can't be fetched from Deepstore?
x
are you running your own data center or cloud ?
m
Datacenter
x
I would argue if you ever have the 2 nodes down for a 100 nodes cluster regularly, then you need to get the data center team to fix the SLA first. Otherwise from software side, you need to ensure data with 3 replicas to eat the inefficiency from them.
m
Agreed we can work on that. But trying to understand where deep store is helpful ? If this is the case why do i need to maintain deepstore infra and just live with pinotservers and replica considering max nodes going down sla.
x
it’s for data backup and distribution to server. There is a mode of P2P which allow you to run pinot without deep store, but we don’t recommend that as that was built for the low availability deep store.
m
1. if its for data backup => how is this not available when required? or auto pulling from deepstore. 2. distribution to pinot server from deepstore ; when pinot server is back up with segments already exist on pinot? Then only it works? Pls correct my understanding. Im trying to make my tech management to understand why hdfs is required? If this is not auto-pulling or manual command that makes you to pull from hdfs to pinot, then the disclaimer for keep deepstore mandatory is being challenged.
x
1. deep store has its own availability. It only serves as the data backup, and on the critical data query path. 2. In order to guarantee the query SLA, Pinot requires data to be on servers to serve queries 3. Since usually there would be 100G to TB data on server, there shouldn’t be auto data movement between servers, that will easily causing avalanche and break the whole system. 4. If users disagreed with 3 and want to move data all the time, then it’s users responsibility to trigger the data movement by the rebalance API, and it will impact the query SLA.