Hi I m trying to set up Pinot for the first time as a cluste Apache Pinot #troubleshooting

Hi! I'm trying to set up Pinot for the first time ...

Aaron Wishnick

02/10/2021, 5:29 PM

Hi! I'm trying to set up Pinot for the first time (as a cluster) and am starting to set up S3 following the steps in https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3. It works fine for the controller, but for the server I see this error on startup:

Copy code

2021/02/10 12:22:25.506 ERROR [PluginManager] [main] Failed to load plugin [pinot-s3] from dir [/<redacted>/apache-pinot-incubating-0.6.0-bin/plugins/pinot-file-system/pinot-s3]                             
java.lang.IllegalArgumentException: object is not an instance of declaring class

Xiang Fu

02/10/2021, 5:33 PM

is there more stacktrace from server side ?

Aaron Wishnick

02/10/2021, 5:34 PM

Sure, I'll paste more

Xiang Fu

02/10/2021, 5:34 PM

Is it during server startup or when you upload data to the table

Aaron Wishnick

02/10/2021, 5:34 PM

During server startup

Aaron Wishnick

02/10/2021, 5:34 PM

Copy code

2021/02/10 12:22:25.506 ERROR [PluginManager] [main] Failed to load plugin [pinot-s3] from dir [/<redacted>/apache-pinot-incubating-0.6.0-bin/plugins/pinot-file-system/pinot-s3]                             java.lang.IllegalArgumentException: object is not an instance of declaring class                                
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]                                  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]        
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]                                              
        at org.apache.pinot.spi.plugin.PluginClassLoader.<init>(PluginClassLoader.java:50) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                          
        at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:171) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                      
        at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                                   
        at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137) [pinot-all-0.6.0-jar-with-depe
ndencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                                            at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103) [pinot-all-0.6.0-jar-with-depe
ndencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                                            at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84) [pinot-all-0.6.0-jar-with-dep
endencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                                           at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) [pinot-all-0.6.0-jar-with-d
ependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                                 
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:164) [pinot-all-0.6.0-ja
r-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]

Aaron Wishnick

02/10/2021, 5:35 PM

I see a report of a similar issue with GCS here: https://lists.apache.org/x/thread.html/rb5e000b1e81b7e44c98bc4bdec03fe89b3113bc93cd7b318378db1ac@%3Cdev.pinot.apache.org%3E

Xiang Fu

02/10/2021, 5:36 PM

ic, are you using the docker image or the baremetal

Xiang Fu

02/10/2021, 5:36 PM

with bin

Aaron Wishnick

02/10/2021, 5:37 PM

Baremetal

Xiang Fu

02/10/2021, 5:37 PM

can you try with java opts to only include the plugins you are using:

-Dplugins.include=pinot-s3,pinot-parquet

Xiang Fu

02/10/2021, 5:37 PM

something like this

Xiang Fu

02/10/2021, 5:37 PM

Xiang Fu

02/10/2021, 5:38 PM

which java version are you using ?

Aaron Wishnick

02/10/2021, 5:38 PM

Yeah, I'm already doing that with java opts

Aaron Wishnick

02/10/2021, 5:38 PM

java 11

Xiang Fu

02/10/2021, 5:38 PM

Aaron Wishnick

02/10/2021, 5:38 PM

openjdk version "11.0.9.1" 2020-11-04

Xiang Fu

02/10/2021, 5:38 PM

hmm

Xiang Fu

02/10/2021, 5:39 PM

I think for java11, we are trying to include all the plugins into classpath as some classloader apis are deprecated from java 8

Xiang Fu

02/10/2021, 5:39 PM

if that doesn’t work just delete the un-used plugins from pinot-bin directory

Xiang Fu

02/10/2021, 5:39 PM

like pinot-gcs

Aaron Wishnick

02/10/2021, 5:43 PM

But I think this error message is specifically about pinot-s3

Aaron Wishnick

02/10/2021, 5:44 PM

Copy code

Failed to load plugin [pinot-s3] from dir [/scratch/aaron/p
rojects/pinot/apache-pinot-incubating-0.6.0-bin/plugins/pinot-file-system/pinot-s3]

Aaron Wishnick

02/10/2021, 5:48 PM

Ok, I deleted every plugin except for pinot-s3, and I still see the same error message.

Xiang Fu

02/10/2021, 5:48 PM

Xiang Fu

02/10/2021, 5:49 PM

hmm

Xiang Fu

02/10/2021, 5:50 PM

it might be issue of jdk

Xiang Fu

02/10/2021, 5:51 PM

i will check this jdk 11.0.9.1?

Aaron Wishnick

02/10/2021, 5:51 PM

Is pinot not compatible with jdk11?

Aaron Wishnick

02/10/2021, 5:51 PM

Thanks

Xiang Fu

02/10/2021, 5:51 PM

it should

Aaron Wishnick

02/10/2021, 5:51 PM

If it helps:

Copy code

$ java -version
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Debian-1deb10u2, mixed mode, sharing)

Tamás Nádudvari

02/10/2021, 5:53 PM

We also ran into this error with the Java 11 docker images. It might be relevant that the component starts with this warning:

Copy code

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.pinot.spi.plugin.PluginClassLoader (file:/opt/pinot/lib/pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.pinot.spi.plugin.PluginClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

We decided to fall back to a jdk8-based docker image for now.

Xiang Fu

02/10/2021, 5:53 PM

sure, I will take a look, one thing you can try is to build pinot from src code with your current java version

Xiang Fu

02/10/2021, 5:55 PM

right for java 11+ need to add

--add-exports java.base/jdk.internal.ref=ALL-UNNAMED

into JAVA_OPTS

Xiang Fu

02/10/2021, 5:56 PM

it’s different apis for class loader

Xiang Fu

02/10/2021, 5:56 PM

those warnings are related to this behavior @Tamás Nádudvari just fyi

👍 1

Aaron Wishnick

02/10/2021, 6:15 PM

Is Java 11 actually supported or should I downgrade to Java 8?

Xiang Fu

02/10/2021, 6:25 PM

it is supported. have you tried adding

-add-exports java.base/jdk.internal.ref=ALL-UNNAMED

Aaron Wishnick

02/10/2021, 6:30 PM

I tried adding that

Aaron Wishnick

02/10/2021, 6:30 PM

I still see the same error, and I also see the "reflective access" warnings still

Xiang Fu

02/10/2021, 6:43 PM

Xiang Fu

02/10/2021, 6:43 PM

let me check then

Elon

02/11/2021, 3:47 AM

Hey, just seeing this - there is a pr for java11, we use it in prod and it has been working great (g1gc behavior much better, more stable, etc.):

Elon

02/11/2021, 3:47 AM

https://github.com/apache/incubator-pinot/pull/6424

Elon

02/11/2021, 3:47 AM

It's a simple change, you can even copypaste it:) lmk if you have any issues with that

Xiang Fu

02/11/2021, 3:53 AM

sure I will take a look

Aaron Wishnick

02/11/2021, 4:39 PM

I tried out Java 8 for the time being, and now I run into this error when starting up the server:

Aaron Wishnick

02/11/2021, 4:39 PM

Copy code

2021/02/11 11:37:06.847 ERROR [PinotFSFactory] [Start a Pinot [SERVER]] Could not instantiate file system for class org.apache.pinot.plugin.filesystem.S3PinotFS with scheme s3                                                
java.lang.IllegalArgumentException: null
        at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                       
        at org.apache.pinot.plugin.filesystem.S3PinotFS.init(S3PinotFS.java:83) ~[pinot-s3-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
        at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:54) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                         
        at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:74) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                             
        at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.<init>(SegmentFetcherAndLoader.java:71) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                   
        at org.apache.pinot.server.starter.helix.HelixServerStarter.start(HelixServerStarter.java:316) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                             
        at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:150) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                            
        at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:95) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                               
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]     
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]   
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]           
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]               
2021/02/11 11:37:06.851 ERROR [StartServiceManagerCommand] [Start a Pinot [SERVER]] Failed to start a Pinot [SERVER] at 0.883 since launch
java.lang.RuntimeException: java.lang.IllegalArgumentException                                                 
        at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:58) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                         
        at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:74) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                             
        at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.<init>(SegmentFetcherAndLoader.java:71) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                   
        at org.apache.pinot.server.starter.helix.HelixServerStarter.start(HelixServerStarter.java:316) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                             
        at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:150) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                            
        at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:95) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                               
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]     
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]   
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]           
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]               
Caused by: java.lang.IllegalArgumentException
        at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                       
        at org.apache.pinot.plugin.filesystem.S3PinotFS.init(S3PinotFS.java:83) ~[pinot-s3-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
        at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:54) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]                                         
        ... 9 more

Aaron Wishnick

02/11/2021, 4:47 PM

Figured this one out -- apparently the

region

setting is required

👍 1

Xiang Fu

02/11/2021, 7:39 PM

do you find it from stacktrace?

Aaron Wishnick

02/11/2021, 7:41 PM

No, there's no information in the stack trace to help debug this

Aaron Wishnick

02/11/2021, 7:41 PM

I just guessed until it started working

Xiang Fu

02/11/2021, 7:41 PM

I see. I will double check and make the error msg more clear

Xiang Fu

02/11/2021, 7:41 PM

thanks for reporting this

Open in Slack

Previous Next