Hi Team, Can we use pinot-batch-ingestion jar with spark 3.1.2 ?

I haven’t tested it out on Spark3.1 but we don’t bundle the spark libs in the batch jars and always use env spark libs. IMO, it should work.

This is what we do. It work just fine. Are you using a normal pinot release or building from source?

@Johan Adami I am using normal pinot release

we built from source, so it required

-P build-shaded-jar

. but with the release 3.1.2 should be fine

Okay, are you using Pinot 0.11.0 or Pinot 0.10.0 ?

using latest master as of a few days ago

for the reference, do you mind sharing one of the job.yaml config you are using for your batch ingestion jobs ? Asking this because Pinot docs are using org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner while latest release has spark3 as the class name (if I am not mistaken)

executionFrameworkSpec:
  name: spark
  segmentMetadataPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentMetadataPushJobRunner
  segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentGenerationJobRunner
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentUriPushJobRunner

that’s what the class names look like

thanks @Johan Adami it worked, I am able to write segments using pinot0.11.0 batch ingestion jar using spark 3.1.2