https://pinot.apache.org/ logo
Join Slack
Powered by
It seems to only happen with the jdk11 (both 0.5 a...
# general
v
It seems to only happen with the jdk11 (both 0.5 and 0.6-RC), i tested with jdk8 and it works fine
k
Thanks @vmarchaud . @Elon can you please help here
k
Looks like Noah sweeped in and rescued the day! LOL . Thank you, @Noah Prince !
v
Well he said that doesnt work with his fix anyway
n
The bit that wasn’t working was, I think, a different issue.
You may want to try my fix
I’m trying to run things in spark, and spark is grumpy at best.
v
i'm using the official docker image so i can't recompile it right now
is there a PR that fixes that ?
e
Oh great! We have it working with pinot-0.5.0 but didn't try with 0.6.0-RC, also we're using jdk8 still. Nice fix @Noah Prince!
n
Not yet. I may include it in a different PR I’m working on. Haven’t done a PR to pinot yet, so may be faster if someone else just makes a PR with this small fix.
👍 1
I also haven’t tested that fix with Java 8. And I’m not sure I’d be the best person to verify this fix given I’m pretty inexperienced with Pinot.
Oh… and now that I’m testing it with java 11 in the server starter… It doesn’t work. Yeah this is going to need a bigger fix. I think @Xiang Fu is looking into parts of this issue right now.
e
Maybe try deploying with java8 and see if it works? We use the gcs fs plugin with pinot 0.5.0 + java8
v
@Elon Do you mind if i reach out in private to ask few questions about your setup ?
e
Not at all!
x
@vmarchaud can you show me the command you are launching the ingestion job?
v
Copy code
docker run --rm -ti \
    --network=host \
    --mount type=bind,source="$(pwd)",target=/tmp \
    apachepinot/pinot:0.6.0-SNAPSHOT-ca8545b29-20201105-jdk8 LaunchDataIngestionJob \
    -jobSpecFile /tmp/jobs/gcs_tracking_events_ingestion_job.yml
as i said its working fine with jdk8
x
ok
v
however the error is in the controller
x
so jdk11 image doesnt work with this cmd?
do u have controller config
if you use gcs as deep store, then there are extra configs you need to put to register the pinotfS
this is an example for s3
v
i configured it on the controller like advised:
Copy code
pinot.set.instance.id.to.hostname=true
      controller.data.dir=<gs://pinot-segment-store/>
      controller.enable.split.commit=true
      controller.allow.hlc.tables=false
      <http://pinot.controller.storage.factory.class.gs|pinot.controller.storage.factory.class.gs>=org.apache.pinot.plugin.filesystem.GcsPinotFS
      pinot.controller.storage.factory.gs.projectId=my-project-id
      pinot.controller.storage.factory.gs.gcpKey=/opt/pinot/conf/gcp/key.json
      pinot.controller.segment.fetcher.protocols=file,http,gs
      pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
x
got it, so just for clarification, this works with jdk8 but not jdk11 right?
v
yep
x
Got it, I will check and fix this in java 11
v
thanks a lot !
x
I tried to test with s3 in jdk11 docker image, seems everything goes well on my side, also putting it into a doc: https://docs.pinot.apache.org/users/tutorials/use-s3-and-pinot-in-docker
seems to be issue with gcs plugin
do you have the stacktrace?
v
Yes, here is the stacktrace with `0.5.0`:
Copy code
java.lang.IllegalArgumentException: object is not an instance of declaring class

	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]

	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

	at org.apache.pinot.spi.plugin.PluginClassLoader.<init>(PluginClassLoader.java:50) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]

	at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:171) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]

	at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]

	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]

	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]

	at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]

	at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]

	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:166) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
And here is the one for `0.6.0`:
Copy code
java.lang.IllegalArgumentException: object is not an instance of declaring class

	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]

	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

	at org.apache.pinot.spi.plugin.PluginClassLoader.<init>(PluginClassLoader.java:50) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-bee125efe3eee3c2ef2298168f12b723519fb1af]

	at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:171) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-bee125efe3eee3c2ef2298168f12b723519fb1af]

	at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-bee125efe3eee3c2ef2298168f12b723519fb1af]

	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137) [pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-bee125efe3eee3c2ef2298168f12b723519fb1af]

	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103) [pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-bee125efe3eee3c2ef2298168f12b723519fb1af]

	at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84) [pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-bee125efe3eee3c2ef2298168f12b723519fb1af]

	at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) [pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-bee125efe3eee3c2ef2298168f12b723519fb1af]

	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:166) [pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-bee125efe3eee3c2ef2298168f12b723519fb1af]
x
hmm is this also with image: apachepinot/pinot:0.6.0-SNAPSHOT-ca8545b29-20201105-jdk11
v
I believe i've tested with the commit 
2d446d2e0e5fa0f58398f515e40174c6cba90719
and it wasn't working
(in docker)
but i'm not sure anymore, i'll to re-test
x
ic, i will set up a google cloud account and test with gcs as well
e
lmk if you have any issues
Open in Slack
PreviousNext
Powered by