One more question, folks: when it comes to segments of ~200M in size, what segment storage technology would you recommend using when running a cluster in AWS? HDFS? S3? EFS mounted?

Ebs

Yes, for local storage attached to serving nodes you can use EBS. For deep store you can use S3.

@User - you can also use HDFS for deep store.

Nothing in a presentable form.

Thanks guys, but which one of those options do you think that gives us the best performance, say, in a scenario of having something like up to 10T in data?

What’s your qps and latency expectation

QPS up to a few dozens at max, latency can be seconds but preferably under 1 minute. Thanks for the info, man!

Yeah, you definitely don’t need local SSD for this. As Kishore mentioned, any of the options for network attached disk on serving node will work.

Ah, awesome, thank you guys!

Since the latency is not too tight, you might want to pack a lot of data per instance, so EBS for serving nodes seems good. For deep store, S3 or HDFS both work (S3 is more popular in my personal experience).

Got it. I'll take that into consideration, and also probably go for S3 for the deep store backups (since we already use it a lot for other things)

If you want latency in seconds ideally, have you considered Hudi?

Not really; I'm not sure what role that would play when integrated to Pinot however