Has anyone used/tested Pinot with a 1 PB data set?...
# general
r
Has anyone used/tested Pinot with a 1 PB data set? If not, could Pinot/ZK handle 2M segments given enough resources? @Xiang Fu @Xiaobing
j
1PB yes. 1M segments no
i think 220k is as far as we got in segment count. you can keep kicking the ZK jute maxbuffer higher
but scanning all those segments across hundreds of servers leads to bad tail latencies. and it’s expensive with all that data on disk
r
How large can an individual segment get? My 2M segment estimate was based on the recommended segment size of 500MB per segment. But with a total segment count of 200K for a 1 PB table means that each segment would be 5TB. Are you working with segments in the TB range?
j
is that math right? it should be 5 gb per segment?
we scaled down this use case, so i’m having to search old notes we have. looks like we had closer to 500k segments averaging ~2gb each
and a ZK cluster with ~7gb heap
it definitely can work. but it wasn’t as stable as we would have liked. there was at least 2-3 issues with random ZK failures or unexpected query failures a year. Rebuilding the broker routing map took ~10 minutes. server restarts took 4+ hours either downloading from s3 or rebuilding segments locally because indices had changed
r
> is that math right? it should be 5 gb per segment? Good catch. Yes, with 200K segments that would be 5GB/segment.
x
echo on what Johan shared. couldn't add more from my side. I had table going over 300K segments and experienced those pains.
r
@Xiaobing For your table with 300K segments, what was the average/max segment size?
Do you guys recall how many servers/brokers were used?
@Xiaobing What's the status of ZK removal/replacement?
j
only 3 brokers. but like 700+ servers
x
there are discussions about how to work around ZK bottlenecks, e.g. one option would be table federation, which is like the concept of hybrid table but allows a table to have many internal sub tables to spread the table/segment metadata load to many ZK nodes
r
interesting. is the intent of the sub tables approach to lower/limit the number of segments per table?
j
I think yes. the bottlenecks I’ve seen • idealstate is huge causing ZK communication to fail due to max node size limit • idealstate is huge causing long serde and update times causing cluster operations to time out • zkclient won’t let you list children when there’s > 100k (too many segments problem)
r
sounds like splitting the 1PB table into 5 200TB or 10 100TB tables would be more manageable
j
yes, but that puts the onus on you to implement how to query it. We had this same problem when we had multi-region data in 2 clusters. gotta figure out how to reduce results
k
Please file an issue.. we are working on a few ideas to overcome this limitation
r
Can you share any of those ideas? They might be useful as workarounds
x
from data layout perspective, if your data have timestamp, you may have yearly or monthly table then you can query and manually stitch/union results together
r
It Pinot had support for views, I would union the yearly/monthly tables in a view
k
yeah.. thats a good idea as well. I think the high level idea is split the Idealstate.. the split can happen based on any of the following • time range (dynamic increase of znodes) • hash of segment names (fixed partitioning)
but from the code perspective its still a single ideal state
r
the problem with a view though is that they're static. for example, it would have to be updated every time a yearly/monthly table is created/deleted
actually what we're describing sounds similar to table partitioning in Postgres