Hi Team, I've been reviewing the architecture for the Spark-Pinot connector, specifically from the documentation here. In this architecture, it's indicated that the Spark executors are supposed to connect directly to individual Pinot servers. My current Pinot setup is running on EKS, while the Spark job is running externally (outside of the EKS cluster). However, no LoadBalancer has been created for the Pinot servers, as it’s generally considered unnecessary for this setup. Given that the Pinot servers are only accessible within the internal Kubernetes network, how will the Spark executors, running outside the EKS cluster, be able to connect to the individual Pinot servers? Would it be necessary to expose the Pinot servers via a LoadBalancer, or is there an alternative configuration that I might be missing to facilitate this communication?

In short yes. You need to create your own load balancers to expose every pinot server outside k8s. And you need to modify the spark pinot connector code to handle the internal dns name(pinot-0.default.local) to the external accessible dns.