@Xiang Fu Im facing below error when trying to read pinot data using spark 3 connector, I'm able to successfully print scheme of table, however I'm facing below issue when printing df.show() or df.count().. can someone please help?

Traceback (most recent call last):
  File "/Users/nitheeshkumar.prabakaran/frameworks/pinot4/pinot/pinot-connectors/pinot-spark-3-connector/target/test.py", line 56, in <module>
    print(f"Total rows:{df.count()}")
  File "/Users/nitheeshkumar.prabakaran/frameworks/spark/spark-3.5.8-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1239, in count
  File "/Users/nitheeshkumar.prabakaran/frameworks/spark/spark-3.5.8-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/Users/nitheeshkumar.prabakaran/frameworks/spark/spark-3.5.8-bin-hadoop3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
  File "/Users/nitheeshkumar.prabakaran/frameworks/spark/spark-3.5.8-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o32.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 0.0 failed 1 times, most recent failure: Lost task 4.0 in stage 0.0 (TID 4) (10.191.22.199 executor driver): org.apache.pinot.connector.spark.common.PinotException: 10.250.19.128:7050(OFFLINE) could not respond the query
	at org.apache.pinot.connector.spark.common.reader.PinotServerDataFetcher.fetchData(PinotServerDataFetcher.scala:146)
	at org.apache.pinot.connector.spark.common.reader.PinotAbstractPartitionReader.getIteratorAndSource(PinotAbstractPartitionReader.scala:65)
	at org.apache.pinot.connector.spark.common.reader.PinotAbstractPartitionReader.$init$(PinotAbstractPartitionReader.scala:39)
	at org.apache.pinot.connector.spark.v3.datasource.PinotScan$$anon$1.<init>(PinotScan.scala:85)
	at org.apache.pinot.connector.spark.v3.datasource.PinotScan.$anonfun$createReaderFactory$1(PinotScan.scala:85)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.advanceToNextIter(DataSourceRDD.scala:92)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:71)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

how are you connecting ? also are pinot components port open to your spark cluster ?

I'm running both pinot and spark locally on pc

How do you configure the connection?

I'm just spinning up the pinot locally and passing spark connector jars to spark submit command which in turn submit pyspark code

yes, Pinot controller