Hi, I was trying to use spark to do batch ingestion. From this doc https://docs.pinot.apache.org/users/tutorials/ingest-parquet-files-from-s3-using-spark, it seems Pinot support Spart 2.X version at least for Pinot 0.4. it seems there is some dependency issue when I was using Spark 2.2.3 like the tutorial but I was able to use Spark 2.4.8 to do the ingestion. Since latest version of Spark is 3.X and Pinot is already 0.8, I'm wondering what's current recommended compatible Spark version ?

@User ^^

@User I would recommend for now staying with Spark 2.x version which has been well tested with latest Pinot versions, there are some known issues w.r.t Spark 3 that team is still working on. Once those are addressed we will update docs to indicate spark 3.x support.

@User Thanks for clarification !