@User - I’m wondering how thoroughly the classloader change (for Java9+ compatibility) was tested with Hadoop/Spark segment building code, when running with Java 8. Asking because we (and others) have run into class-not-found issues (for plugins) when trying to build segments with 0.8.0 using Hadoop and Spark on Java 8, and I see this change to

PluginClassLoader.java

that I think you made:

< import java.lang.reflect.Method;
39,47d37
<     Method method = null;
<     try {
<       method = URLClassLoader.class.getDeclaredMethod("addURL", URL.class);
< 
<     } catch (NoSuchMethodException e) {
<       //this should never happen
<       ExceptionUtils.rethrow(e);
<     }
<     method.setAccessible(true);
50c40,47
<         method.invoke(classLoader, url);
---
>         /**
>          * ClassLoader in java9+ does not extend URLClassLoader.
>          * If the class is not found in the parent classloader,
>          * it will be found in this classloader via findClass().
>          *
>          * @see <https://community.oracle.com/tech/developers/discussion/4011800/base-classloader-no-longer-from-urlclassloader>
>          */
>         addURL(url);

@User I think you wrote the original code here, right?

We don't use spark for ingestion here (use direct insert from trino - have an open pr) so I did not test it. Can you create a github issue - I would be happy to help.

Hi @User - thanks for any help. Issue is at https://github.com/apache/pinot/issues/7393

yes, this needs to change with all the changes that happened in jvm

Hi @User @User I built a version of 0.8.0 that reverted the change to

PluginClassLoader.java

, and the Hadoop job now runs w/o errors. So it seems pretty clear this was the change that broke things for Hadoop/Spark jobs. So…we could add a Java version check (kind of gross), or is there some approach that’s compatible with both Java 8 and Java 9+?

I remember breaking my head trying to get it to work with various versions without a version check… if you can find something, happy to review

hmm, then shell we consider do multi-version releases for this class?

I’m not clear why Java 8 requires the old code (which uses reflection), as for both Java 8 and Java 9 the URLClassloader.addUrl() method is protected. So I would expect @User’s version to work fine for both, but it clearly doesn’t.