Hi all, I am using pinot-offline ingestion to upl...
# troubleshooting
r
Hi all, I am using pinot-offline ingestion to upload data from Hdfs Cluster A into Pinot backed by Hdfs cluster B using spark. The hadoop config is read from the cluster on which the spark job is running. Is there a way to pass Hdfs config of both the clusters together... Or any other approach that you suggest??
m
Afaik, there can only be one PinotFS configured at a time. Any reason for using two different ones?
r
We need to run the offline job on cluster A which has the data due to data access policy and then ingest into pinot which is backed hdfs B...
m
One optimization you could do is if you can copy the Pinot segments generated by Spark Job into HDFS B and then just do a metadata push into Pinot.
This way the segment payloads won't need to go through the controller.
r
Understood... thanks a lot 🙏