https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • v
    Uploading segment dauSegments_52.tar.gz Sending request: http://x.x.x.x:9000/v2/segments to controller: ip-.ec2.internal, version: 0.1.0-79de69d37bc777962853cff55ed0e2397675c7b0 Compressing segment dauSegments_95 Uploading segment dauSegments_95.tar.gz Sending request: http://x.x.x.x:9000/v2/segments to controller: ip-.ec2.internal, version: 0.1.0-79de69d37bc777962853cff55ed0e2397675c7b0 Compressing segment dauSegments_144
  • v
    sample logs from segments being uploaded
  • v
    i had uploaded all of them
  • v
    not sure if I am missing some config settings to look for all the segments while querying
  • k
    will ping you on a separate channel to get some additional details
  • v
    Exception in thread "main" java.lang.NullPointerException at org.apache.pinot.hadoop.job.BaseSegmentJob.getDataFilePathsHelper(BaseSegmentJob.java:71) at org.apache.pinot.hadoop.job.BaseSegmentJob.getDataFilePaths(BaseSegmentJob.java:65) at org.apache.pinot.hadoop.job.SegmentCreationJob.run(SegmentCreationJob.java:127) at org.apache.pinot.hadoop.PinotHadoopJobLauncher.kickOffPinotHadoopJob(PinotHadoopJobLauncher.java:50) at org.apache.pinot.hadoop.PinotHadoopJobLauncher.main(PinotHadoopJobLauncher.java:90) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) this is while reading s3 data using hadoop jar
  • v
  • v
    <!here> i was trying to create segment from s3 to s3
  • x
    @User could you try to put hadoop-s3 jar into the classpath
  • x
    hadoop-aws-2.7.0.jar
  • x
    you can find it under 
    pinot-distribution/target/apache-pinot-incubating-0.2.0-SNAPSHOT-bin/apache-pinot-incubating-0.2.0-SNAPSHOT-bin/lib
    once you build the distribution
  • s
    Hi Xiang, im deploying kafka in azure kubernetes (AKS) .. it is failing to intialize the initi container
  • s
  • s
    sundar:skaffold $ kubectl get pods -n pinot-quickstart | grep kafka kafka-0 0/1 Init:CrashLoopBackOff 5 5m12s sundar:skaffold $
  • x
    will pm you
  • k
    If you are at GHC, dont miss the Pinot talk from @User. She will talk about the secrets behind Pinot performance. cc @User and @User will also be there if you want to know more about Apache Pinot.
    👍 3
  • h
    Could you share the details to create/upload the segments in parallel instead of sequential?
  • s
    for parallelizing segment create/upload, we currently provide the hadoop job. https://pinot.readthedocs.io/en/latest/pinot_hadoop.html#creating-segments-using-hadoop
  • j
    Hi guys, I am trying to run the offline quick-start demo with compiled code on my macbook, but I get IllegalStateException:
    Copy code
    LM-SHB-24502350:apache-pinot-incubating-0.2.0-SNAPSHOT-bin junstan$ bin/quick-start-offline.sh
***** Starting Zookeeper, controller, broker and server *****
Executing command: StartZookeeper -zkPort 2123 -dataDir /var/folders/mn/8gp0gxgx24g5n05wmkcbk1r03khl7m/T//PinotAdmin/zkData
Start zookeeper at localhost:2123 in thread main
Executing command: StartController -clusterName QuickStartCluster -controllerHost 192.168.1.3 -controllerPort 9000 -dataDir /var/folders/mn/8gp0gxgx24g5n05wmkcbk1r03khl7m/T//PinotController -zkAddress localhost:2123
Invalid instance setup, missing znode path: /QuickStartCluster/CONFIGS/PARTICIPANT/Controller_192.168.1.3_9000
Invalid instance setup, missing znode path: /QuickStartCluster/INSTANCES/Controller_192.168.1.3_9000/MESSAGES
Invalid instance setup, missing znode path: /QuickStartCluster/INSTANCES/Controller_192.168.1.3_9000/CURRENTSTATES
Invalid instance setup, missing znode path: /QuickStartCluster/INSTANCES/Controller_192.168.1.3_9000/STATUSUPDATES
Invalid instance setup, missing znode path: /QuickStartCluster/INSTANCES/Controller_192.168.1.3_9000/ERRORS
1 nodes of type INSTANCE were requested but the tree has only 0 nodes!
Event 96131382_DEFAULT : Error computing assignment for resource leadControllerResource. Skipping.
java.lang.IllegalStateException: null
        at org.apache.helix.controller.rebalancer.strategy.crushMapping.CRUSHPlacementAlgorithm$Selector.select(CRUSHPlacementAlgorithm.java:308) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.strategy.crushMapping.CRUSHPlacementAlgorithm.select(CRUSHPlacementAlgorithm.java:119) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy.doSelect(CrushRebalanceStrategy.java:173) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy.select(CrushRebalanceStrategy.java:140) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy.computePartitionAssignment(CrushRebalanceStrategy.java:92) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy.computePartitionAssignment(CrushRebalanceStrategy.java:48) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.strategy.AbstractEvenDistributionRebalanceStrategy.computePartitionAssignment(AbstractEvenDistributionRebalanceStrategy.java:88) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.strategy.AbstractEvenDistributionRebalanceStrategy.computePartitionAssignment(AbstractEvenDistributionRebalanceStrategy.java:49) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.AutoRebalancer.computeNewIdealState(AutoRebalancer.java:128) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.rebalancer.AutoRebalancer.computeNewIdealState(AutoRebalancer.java:51) ~[helix-core-0.8.4.jar:0.8.4]
        at org.apache.helix.controller.stages.BestPossibleStateCalcStage.computeResourceBestPossibleState(BestPossibleStateCalcStage.java:244) ~[helix-core-0.8.4.jar:0.8.4]
    .
  • j
    Could someone help me please?
  • k
    @User It’s not a fatal exception. @User is working on fixing it.
  • k
    You should be able to access the query console.
  • j
    Thanks @User, I can start the demo now.
  • j
    But I still cannot run a query in the console. Here is the exception message:
  • j
    Copy code
    ***** Waiting for 5 seconds for the server to fetch the assigned segment *****
***** Offline quickstart setup complete *****
Total number of documents in the table
Query : select count(*) from baseballStats limit 0
Executing command: PostQuery -brokerHost 192.168.1.3 -brokerPort 8000 -query select count(*) from baseballStats limit 0
Exception in thread "main" java.lang.NullPointerException
        at org.apache.pinot.tools.Quickstart.prettyPrintResponse(Quickstart.java:76)
        at org.apache.pinot.tools.Quickstart.execute(Quickstart.java:177)
        at org.apache.pinot.tools.Quickstart.main(Quickstart.java:209)
Oct 08, 2019 9:59:20 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: HK2 service reification failed for [javax.servlet.ServletConfig] with an exception:
MultiException stack 1 of 2
java.lang.NoSuchMethodException: Could not find a suitable constructor in javax.servlet.ServletConfig class.
        at org.glassfish.jersey.inject.hk2.JerseyClassAnalyzer.getConstructor(JerseyClassAnalyzer.java:168)
        at org.jvnet.hk2.internal.Utilities.getConstructor(Utilities.java:156)
        at org.jvnet.hk2.internal.ClazzCreator.initialize(ClazzCreator.java:105)
        at org.jvnet.hk2.internal.ClazzCreator.initialize(ClazzCreator.java:156)
        at org.jvnet.hk2.internal.SystemDescriptor.internalReify(SystemDescriptor.java:716)
        at org.jvnet.hk2.internal.SystemDescriptor.reify(SystemDescriptor.java:670)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:441)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2287)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.igdCacheCompute(ServiceLocatorImpl.java:1163)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.access$400(ServiceLocatorImpl.java:105)
        at org.jvnet.hk2.internal.ServiceLocatorImpl$8.compute(ServiceLocatorImpl.java:1157)
        at org.jvnet.hk2.internal.ServiceLocatorImpl$8.compute(ServiceLocatorImpl.java:1154)
        at org.glassfish.hk2.utilities.cache.internal.WeakCARCacheImpl.compute(WeakCARCacheImpl.java:105)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(ServiceLocatorImpl.java:1237)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetInjecteeDescriptor(ServiceLocatorImpl.java:558)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getInjecteeDescriptor(ServiceLocatorImpl.java:567)
        at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.lambda$new$0(ContextInjectionResolverImpl.java:81)
        at org.glassfish.jersey.internal.util.collection.Cache$OriginThreadAwareFuture.lambda$new$0(Cache.java:169)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.jersey.internal.util.collection.Cache$OriginThreadAwareFuture.run(Cache.java:225)
        at org.glassfish.jersey.internal.util.collection.Cache.apply(Cache.java:77)
        at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.resolve(ContextInjectionResolverImpl.java:95)
        at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.resolve(ContextInjectionResolverImpl.java:121)
        at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueParamProvider.lambda$getValueProvider$0(DelegatedInjectionValueParamProvider.java:67)
        at org.glassfish.jersey.server.spi.internal.ParamValueFactoryWithSource.apply(ParamValueFactoryWithSource.java:50)
        at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:64)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:109)
        at
  • k
  • k
  • k
    can you open this?
  • j
    yes, I am making the query on this website.
  • k
    yeah, sometimes the cluster does not get setup within 5 seconds and we see that error. Let me fix that
1...878889...160Latest