Hi How would I go about creating a custom filesystem plugin Apache Pinot #troubleshooting

Join Slack

Hi, How would I go about creating a custom filesys...

# troubleshooting

Yash Agarwal

12/22/2020, 1:40 PM

Hi, How would I go about creating a custom filesystem plugin ?

Ken Krugler

12/22/2020, 3:22 PM

Hi @Yash Agarwal - you should be able to fork the pinot repo from https://github.com/apache/incubator-pinot, then add your own plugin directory at

<repo>/pinot-plugins/pinot-file-system/<your plugin>

. It would automatically get picked up if it’s located in the

plugins/

directory inside of a Pinot distribution.

Ken Krugler

12/22/2020, 3:24 PM

You’d add your plugin module name to the list of modules in the

…/pinot-file-system/pom.xml

file so it gets built. I believe it’s also possible to code this up completely separate from the Pinot project, but I haven’t tried that (ie just a regular project with appropriate dependencies on Pinot jars)

Ken Krugler

12/22/2020, 3:27 PM

And at the code level, you’re implementing a class that extends

PinotFS

Yash Agarwal

12/22/2020, 5:15 PM

yeah. I was hoping if it could be a separate code. We already our working on the way you mentioned. Thanks!

Yash Agarwal

12/22/2020, 5:19 PM

Btw

Copy code

-Dplugins.dir=/opt/pinot/plugins

we provide this path in our docker image for picking plugins but I get exceptions when loading plugins.

Ken Krugler

12/22/2020, 6:19 PM

@Yash Agarwal - what exceptions? And for regular plugins, or your custom plugin?

vmarchaud

12/30/2020, 4:41 PM

Just wanted to add: we've succesfully wrote our own plugin for realtime ingestion, dont hesitate if you hit a wall

Yash Agarwal

01/04/2021, 7:35 AM

We get the following exception when starting our pinot nodes. with config

-Dplugins.dir=/opt/pinot/plugins

Copy code

Failed to load plugin [pinot-hdfs] from dir [/opt/pinot/plugins/pinot-file-system/pinot-hdfs]
java.lang.IllegalArgumentException: object is not an instance of declaring class
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.pinot.spi.plugin.PluginClassLoader.<init>(PluginClassLoader.java:50) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-85cf696be3decb75fd8e0d9e5ec5ef0a32d8dd9b]
	at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:171) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-85cf696be3decb75fd8e0d9e5ec5ef0a32d8dd9b]
	at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-85cf696be3decb75fd8e0d9e5ec5ef0a32d8dd9b]
	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-85cf696be3decb75fd8e0d9e5ec5ef0a32d8dd9b]
	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-85cf696be3decb75fd8e0d9e5ec5ef0a32d8dd9b]
	at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-85cf696be3decb75fd8e0d9e5ec5ef0a32d8dd9b]
	at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-85cf696be3decb75fd8e0d9e5ec5ef0a32d8dd9b]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:166) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-85cf696be3decb75fd8e0d9e5ec5ef0a32d8dd9b]

Xiang Fu

01/05/2021, 1:44 AM

hmm, is there any change ?

Xiang Fu

01/05/2021, 1:44 AM

if it’s java 11, then you don’t need to specify this

-Dplugins.dir=/opt/pinot/plugins

Xiang Fu

01/05/2021, 1:44 AM

we should put the jar into classpath

Xiang Fu

01/05/2021, 1:45 AM

this classloader stuff only works on jdk8

Kishore G

01/05/2021, 3:28 AM

@Yash Agarwal was this resolved?

Yash Agarwal

01/07/2021, 5:06 PM

Yes. It is working. Thanks!!

Open in Slack

Previous Next