Hi! I'm planning to use Pinot with a single real-time upsert table that will have around 8b rows and almost all of them will be updated daily with upserts every 30 days. No offline table here as I need to be able to upsert any record at any time, is this an intended usecase for Pinot? I'm asking because the docs always mention a low retention for real-time table, then move segments to offline table (and segment storage). Are there any issues with using Pinot this way? Thank you!

In this cases you need to do capacity planning before hand and give enough servers and kafka partitions. Key space is 8b, what about the total events ?

Total events is around 20b monthly. The amount of kafka partitions is important for throughput or does it affect Pinot in any other way?

Disk space is the other big issue we ran into with this use case. It affects your ability to scale. Every segment for a partition has to live on one server, so you want to pick a highly composite number of partitions. This way, you have lots of options to scale. If your number of servers isn’t a divisor of the number of partitions, then some servers will always hold more segments

@Johan Adami they can run compaction job daily or weekly that will purge old records right?

yes they can! Docs are here. We haven’t set this up yet, and it requires some work, so was just pointing it out

Yeah, disk space is an issue, with my initial tests I'm forecasting I will need around 70TB of disk space, which is a LOT. Ingesting the initial dataset also takes a lot of time with Kafka. I was trying to figure out some strategy to use a realtime table and every N days detect the offline segments I would need to rebuild/create and replace them from the source data I have stored in S3, achieving upsert functionality but in an offline table

70TB is quite a lot. make sure to set any dimension and metric columns to

noDictionaryColumns

. That got our table from ~130TB to ~50TB

Yeah, most columns have low cardinality (less than 5 distinct values) but one column is MV, an array of 2k integers on average (out of 500k possible integers) and I need an inverted index there so it has to be dictionary 😞