Is there any tar or binary file available with pin...
# generaln
Nikhil Thomas
10/31/2023, 8:43 AMIs there any tar or binary file available with pinot jdk 8 compiled one ? When I am using spark ingestion job, pinot is getting failed with unsupported class version error
Nikhil Thomas
10/31/2023, 8:45 AMIMG_20231031_141410__01.jpg
Nikhil Thomas
11/01/2023, 1:53 AMAny help
Nikhil Thomas
11/01/2023, 1:58 AM@Hubert Dulay
h
Hubert Dulay
11/01/2023, 2:06 AMyou can try Pinot 0.3.0 - i quickly scanned the released notes and this was the only one referencing J 8
n
Nikhil Thomas
11/01/2023, 2:13 AMBut how to solve the issue , unsupportedclass version.
In my organization,many of the dependency is not getting downloaded when I do compile the source .
Nikhil Thomas
11/01/2023, 2:17 AM@Hubert Dulay
h
Hubert Dulay
11/01/2023, 1:02 PMWhat was the command you ran?
n
Nikhil Thomas
11/01/2023, 1:22 PMSpark-submit command for segment creation
Nikhil Thomas
11/01/2023, 1:23 PMspark-submit //
--class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand //
--master local --deploy-mode client //
--conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins" //
--conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" //
-conf "spark.executor.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" //
local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar -jobSpecFile /path/to/spark_job_spec.yaml
Nikhil Thomas
11/01/2023, 1:23 PMFollowed this page @Hubert Dulay https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark
h
Hubert Dulay
11/01/2023, 1:25 PMHave you tried using this recipe? https://github.com/startreedata/pinot-recipes/tree/main/recipes/ingest-parquet-files-from-s3-using-spark
Hubert Dulay
11/01/2023, 1:28 PMAs for the error, I would suggest downloading an earlier version to see if the class version error goes away
n
Nikhil Thomas
11/01/2023, 1:29 PMIn my organization,pinot 1.0.0 alredy configured.
h
Hubert Dulay
11/01/2023, 1:30 PMHave you tried using java 11?
n
Nikhil Thomas
11/01/2023, 1:30 PMNo...
Nikhil Thomas
11/01/2023, 1:31 PMBut in the FAQ , it was written like this
Nikhil Thomas
11/01/2023, 1:31 PMScreenshot_20231101-190115.jpg
h
Hubert Dulay
11/01/2023, 1:32 PMSo you built pinot 1.0 using java 8?
n
Nikhil Thomas
11/01/2023, 1:34 PMLot of dependency is required and many of them is not present in my organization artifactory .so build is getting failed.Hence I am asking is there any way get the jav8 compiled pinot 1.0.0 is available
h
Hubert Dulay
11/01/2023, 1:36 PMAny Java8 compiled Pinot wouldn't be sourcing dependencies from you company's artifactory. It sounds like only using libraries from your artifactory is allowed.
p
Peter Pringle
11/02/2023, 2:29 PMUsing end of life Java versions and old library versions with vulnerabilities doesn't seem like a good idea. Usually artifactory will pull missing versions from remote maven repos, or you should have an onboarding worlflow in your organisation to pull in external tar.gz distros/docker images.
n
Nikhil Thomas
11/03/2023, 2:31 AMThen how will be using yarn cluster configured with java8
3 Views