Ken Krugler

05/14/2021, 10:24 PM
I’ve been fooling around with how Pinot handles the “URI push” of segments. It seems like if I’m not using HDFS for deep storage, then the controller will download the segments before pushing to the server, which seems like it’s not a win. Is that correct? And (so far) I haven’t been able to configure the controller to successfully handle an HDFS URI push request, at least when I’m not using HDFS for deep storage - I see the msg when the controller starts up that the “hdfs file system” was initialized, but when it gets the URI push request, it fails with an error about the hdfs file system not being initialized. Any ideas?


05/14/2021, 10:31 PM
URI push should work for all deep-storages that provide uri based access (HDFS/ADLS/GCP/S3), only exception is NFS I'd think
Unsure about why you are seeing that behavior, would need more debugging.

Ken Krugler

05/14/2021, 10:33 PM
I was trying to figure out if you could do an HDFS URI push without enabled deep storage for the same. So instead of pushing actual segments through the controller to be stored locally by server processes, you’d push the URI and the server process would download locally. Sounds like that’s not supported.

Kishore G

05/14/2021, 11:01 PM
you need to use URI with metadata push

Xiang Fu

05/15/2021, 5:08 AM
right, metadata push might be what you want to take a look
it will bypass controller to download the segment
and only server will fetch the segment from deepstore