https://pinot.apache.org/ logo
Join Slack
Powered by
Having some issues getting a custom StreamMessageD...
# general
n
Having some issues getting a custom StreamMessageDecoder plugin to load, <thread> as not to clog up the channel.
Running Pinot 0.5.0 in a docker-compose. I’m trying to write a custom 
StreamMessageDecoder
given we use a binary wire format in kafka. I made and assembled a jar, then put that as a volume into 
/opt/pinot/plugins/my-plugin-assembly-1.0.0.jar
I then boot up the docker-compose and add the table, and it gives a 
java.lang.ClassNotFoundException: my.custom.RecordDecoder
when attempting to create my table. I exec into the container and can see my class in the jar inside 
/opt/pinot/plugins
with `jar -tf.`I can see a bunch of logs at the start of the server saying Using <whatever>, and my jar is not included in that list. The closest exception that seems like it might be relevant is:
Copy code
java.lang.ClassCastException: java.net.URL cannot be cast to java.lang.Comparable
	at java.util.ComparableTimSort.countRunAndMakeAscending(ComparableTimSort.java:320) ~[?:1.8.0_265]
	at java.util.ComparableTimSort.sort(ComparableTimSort.java:188) ~[?:1.8.0_265]
	at java.util.Arrays.sort(Arrays.java:1246) ~[?:1.8.0_265]
	at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:170) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
	at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
	at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
	at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:166) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
Is there a step I missed on how to make it load the classes from my custom plugin? Perhaps compiling the jar with the wrong version of java? I used OpenJDK 1.8.0_265
My docker-compose, if that helps
docker-compose.yaml
x
Can you try to copy your jar into the Pinot plugin directory ?
Also try to mount your plugin to a subdirectory of plugins directory
There are other plugins that Pinot depends on
n
It's directly in the plugin directory. I can try to put it in a subdir and see if that helps
Is it this?
There a good way to enable INFO level logs?
Copy code
environment:
      JAVA_OPTS: -Dlog4j.logger.org.apache.pinot.spi=DEBUG
Nice.
https://github.com/apache/incubator-pinot/blob/release-0.5.0/pinot-spi/src/main/java/org/apache/pinot/spi/plugin/PluginManager.java#L123 Doesn’t seem to ever log that it’s Trying to load plugins, loading all plugins, what have you
Could it be that I generated the jar from scala. Should be interoperable since it’s JVM, no?
Copy code
org.reflections.ReflectionsException: could not get type for name scala.runtime.AbstractFunction0
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.reflections.Reflections.<init>(Reflections.java:126) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:276) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.swagger.jaxrs.config.BeanConfig.scanAndRead(BeanConfig.java:240) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.swagger.jaxrs.config.BeanConfig.setScan(BeanConfig.java:221) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.server.starter.helix.AdminApiApplication.setupSwagger(AdminApiApplication.java:104) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.server.starter.helix.AdminApiApplication.start(AdminApiApplication.java:86) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.server.starter.helix.HelixServerStarter.start(HelixServerStarter.java:401) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:147) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:94) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:200) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:195) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:166) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.tools.admin.command.StartServerCommand.execute(StartServerCommand.java:137) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
Caused by: java.lang.ClassNotFoundException: scala.runtime.AbstractFunction0
Though this looks bad.
Copy code
java.lang.ClassNotFoundException: org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory
Woohoo, got a new failure now
x
Hmm, then seems that the plugins are not loaded
Even if it’s in scala, we should still find the class
Open in Slack
PreviousNext
Powered by