https://pinot.apache.org/ logo
Join Slack
Powered by
Hey, i'm trying to test the master branch of pinot...
# general
v
Hey, i'm trying to test the master branch of pinot but it seems that the build isn't working by default. The new index spi doesn't register indexes by default
Copy code
Aug 18 10:06:02 gha-runner-2 bash[2175298]: Caused by: java.lang.IllegalArgumentException: Unknown index id: dictionary
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.segment.spi.index.IndexService.get(IndexService.java:144) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.segment.spi.index.StandardIndexes.dictionary(StandardIndexes.java:88) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.enableMetricsAggregationIfPossible(MutableSegmentImpl.java:1070) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:247) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.<init>(LLRealtimeSegmentDataManager.java:1462) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:447) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:249) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:168) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:83) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-041a6044afa85da81f457b324a4f9fc22e0475fb]
Aug 18 10:06:02 gha-runner-2 bash[2175298]:         ... 12 more
I've tried to play around with the maven build without luck yet, i'm compiling pinot with the same command as the dockerfile:
Copy code
mvn install package -DskipTests -Pbin-dist -Pbuild-shaded-jar -Djdk.version=11 -T1C
I've checked and the segment-local module is correctly included in the resulting jar but for some reasons indexes plugin aren't registered
m
@Gonzalo Ortiz
g
that seems strange. The CI tests are passing without issues (although they use pinot-tool, so they use a shaded version)
oh, you are using the shaded version as well
which java version are you using?
v
i'm not sure if it's linked the build itself since the build does contains the classes
jdk11
i'm building with 11 and running with 11
just in case:
Copy code
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
Eclipse OpenJ9 VM AdoptOpenJDK-11.0.11+9 (build openj9-0.26.0, JRE 11 Linux amd64-64-Bit Compressed References 20210421_975 (JIT enabled, AOT enabled)
OpenJ9   - b4cc246d9
OMR      - 162e6f729
JCL      - 7796c80419 based on jdk-11.0.11+9)
but since classes are correctly shaded in the jar, i would have guessed the issue come from the plugins init not working correctly, specially the serviceloader
g
It is not just the classes, the manifest are also important
v
right, is there a way i can help here ?
g
Can you try with another jdk?
v
building or running ?
g
I'm running
Copy code
mvn install package -DskipTests -Pbin-dist -Pbuild-shaded-jar -Djdk.version=11 -T1C
locally to try myself, but I usually use more standard jdks
building or running ?
running
v
i'm running with
Copy code
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)
what do you suggest ? i must admit i'm not sure which one to use ahah
g
btw, what are you doing to test that? just run pinot?
v
trying to ingest rows, either with a realtime table or via the minion
the stacktrace above is for the server
g
btw, adoptjdk is not updated anymore. I would recommend you to upgrade to temurin (same thing, but keeps updated)
I'm going to try to use the same minor version (11.0.11) to try to replicate the issue
v
trying now
g
wow, 11.0.11 is pretty old. I cannot find it in the mainstream channels 😄
v
yeah time flies ahah i think i used it when we initially deployed pinot in 2021
back then that was the latest stable i believe
i'll try to use correto since its the one used by the docker build
g
any other flavor should work... at least in a recent version
v
i've updated my test env with correto and still have the same error
i'm trying to rebuild it now
g
can you do:
Copy code
unzip -p build/lib/pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin
in the pinot git root folder?
or if you already have the distribution somewhere else,
Copy code
unzip -p lib/pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin
in the root folder of that distribution
I'm running 
.github/workflows/scripts/.pinot_quickstart.sh
with the jdk you linked here and it seems to be working
the unzip command I've sent above should print something like:
Copy code
> unzip -p build/lib/pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin 
org.apache.pinot.segment.local.segment.index.bloom.BloomIndexPlugin
org.apache.pinot.segment.local.segment.index.dictionary.DictionaryIndexPlugin
org.apache.pinot.segment.local.segment.index.forward.ForwardIndexPlugin
org.apache.pinot.segment.local.segment.index.fst.FstIndexPlugin
org.apache.pinot.segment.local.segment.index.h3.H3IndexPlugin
org.apache.pinot.segment.local.segment.index.inverted.InvertedIndexPlugin
org.apache.pinot.segment.local.segment.index.json.JsonIndexPlugin
org.apache.pinot.segment.local.segment.index.nullvalue.NullValueIndexPlugin
org.apache.pinot.segment.local.segment.index.range.RangeIndexPlugin
org.apache.pinot.segment.local.segment.index.text.TextIndexPlugin
v
Copy code
caution: filename not matched:  META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin
g
ok... that is bad and may be the reason why we have this behavior
can you unzip that jar? or upload it somewhere so I can analyze it?
specifically I would need to have the META-INF folder
v
sure
pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar
pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar
i'm gonna check the meta-inf with the build with correto when it finishes
same with my new build with correto
not sure if it matters but i'm always using the build inside pinot-distribution
g
the 
build
folder in the root repo should be a soft link to 
pinot-distribution/target/apache-pinot-0.13.0-SNAPSHOT-bin/apache-pinot-0.13.0-SNAPSHOT-bin
v
indeed, i never saw that soft link but yeah we are talking about the same thing
my correto version:
Copy code
openjdk 11.0.20 2023-07-18 LTS
OpenJDK Runtime Environment Corretto-11.0.20.8.1 (build 11.0.20+8-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.20.8.1 (build 11.0.20+8-LTS, mixed mode)
g
Ok
the issue is that you are using the jar with dependencies
at least that is what you have sent here.
That is not the final jar we usually use and in fact it doesn't include shading (which is not related to this issue, but may trigger other issues)
v
not sure how which one to use then ? I'm building it from scratch to include a commit (https://github.com/reelevant-tech/pinot/commit/7eafe96c29a2fcba72fc923a497df624b84a559a)
that's weird because we always have built it like this and use the tar in 
pinot-distribution
when installing in our prod servers
g
the one you should use is 
pinot-distribution/target/apache-pinot-0.13.0-SNAPSHOT-bin/apache-pinot-0.13.0-SNAPSHOT-bin/lib/pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar
, relative to the git root folder
v
that's the one i'm using
building with 
mvn install package -DskipTests -Pbin-dist -Pbuild-shaded-jar -Djdk.version=11 -T1C
here's the one i just built (still doesn't include correct meta-infs)
pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar
g
really? Ok, my bad then.
v
image.png
g
What I'm doing is: 1. mvn clean 2. git checkout master 3. mvn install package -DskipTests -Pbin-dist -Pbuild-shaded-jar -Djdk.version=11 -T1C 4. unzip -p pinot-distribution/target/pinot-distribution-0.13.0-SNAPSHOT-shaded.jar META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin And it returns the list I've sent before
v
this is some kind of sorcery ahah
🤣 1
gonna test with a mvn clean
could you print the jdk you are using ?
g
The forth point was incorrect. I've just fixed it
I'm using
Copy code
openjdk 11.0.19 2023-04-18
OpenJDK Runtime Environment Temurin-11.0.19+7 (build 11.0.19+7)
OpenJDK 64-Bit Server VM Temurin-11.0.19+7 (build 11.0.19+7, mixed mode)
but tbh I wouldn't think the issue is the jdk
v
building ...
i would agree but i don't see why the meta-infs would be different too lol
g
once it finishes, can you do:
Copy code
unzip -p pinot-segment-local/target/pinot-segment-local-0.13.0-SNAPSHOT.jar META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin
from the git root?
v
i can try in my prod env
g
... We are creating that manifest by using autoservice, which is defined as optional to not be added to the final package. Maybe for whatever reason your compilation is deciding to do not include that optional? Can you try to modify 
/pom.xml
(again, in the repo root) and modify the auto-service to remove the optional attribute? It is in line 431 in master
v
they are in the segment-local meta-inf
removing the optional attribut now
re-launching the build
g
they are in the segment-local meta-inf
they are? that is more strange. Then I don't think auto-service is the issue
it should be in the shading process
v
they are there after i stopped my build to remove the optional in local
g
So... let's sync. You have run the following:
Copy code
mvn clean
git checkout master
mvn install package -DskipTests -Pbin-dist -Pbuild-shaded-jar -Djdk.version=11 -T1C
And when you execute
Copy code
unzip -p pinot-segment-local/target/pinot-segment-local-0.13.0-SNAPSHOT.jar META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin
you get
Copy code
org.apache.pinot.segment.local.segment.index.bloom.BloomIndexPlugin
org.apache.pinot.segment.local.segment.index.dictionary.DictionaryIndexPlugin
org.apache.pinot.segment.local.segment.index.forward.ForwardIndexPlugin
org.apache.pinot.segment.local.segment.index.fst.FstIndexPlugin
org.apache.pinot.segment.local.segment.index.h3.H3IndexPlugin
org.apache.pinot.segment.local.segment.index.inverted.InvertedIndexPlugin
org.apache.pinot.segment.local.segment.index.json.JsonIndexPlugin
org.apache.pinot.segment.local.segment.index.nullvalue.NullValueIndexPlugin
org.apache.pinot.segment.local.segment.index.range.RangeIndexPlugin
org.apache.pinot.segment.local.segment.index.text.TextIndexPlugin
but when you execute
Copy code
unzip -p build/lib/pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar  META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin
you get
Copy code
caution: filename not matched:  META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin
right?
v
i'll tell you about 
unzip -p build/lib/pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar  META-INF/services/org.apache.pinot.segment.spi.index.IndexPlugin
when the whole compilation finishes
👍 1
2 Views
Open in Slack
PreviousNext
Powered by