Hi folks, I have some questions about migrating data from a previous database into Pinot: In my project, we'll start publishing data to a Pinot realtime table, but we also need to port historical data. For historical data, do you recommend using an OFFLINE table to be used in conjunction with the REALTIME table, or is it fine to port the historical data to the REALTIME table directly? What are the pros and cons for each approach? Thanks!

You could do either. Typically folks put historical data onto offline table, and just have the RT nodes ingest and serve fresh data

Ah, wait, so the RT nodes should be separate from the OL nodes? They're part of the same cluster, but have to be dedicated for either OL or RT? (Sorry for my ignorance)

Technically you could use the same instances, but in production setup depending on your requirements you may want to keep them separate

Got it. And do you guys have some experience with knowing the performance between storing segments on S3 vs HDFS? How do they compare, which storage do you recommend for an AWS-based cluster?

If you are in AWS, then you could go with S3. Pinot doesn’t really recommend one vs the other. It does require the system to be reliable and available though

Thanks man! And something I've been wondering is, how does it cache segments in memory, if it caches them at all? I'd like to have a better understanding about what I can expect from Pinot in terms of how it decides whether to load segments from deep storage vs reusing cached results or rows

Pinot does not cache query results because it does not make much sense for real-time systems where data is changing. Pinot servers store a local copy of the data on attached disk and serve from there. Deepstore is used by servers to download data in case of data loss or new servers joining the cluster etc

Ah, got it! Thanks!