https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • l
    please help me @Yupeng Fu @Jackie @Xiang Fu
  • j
    Do you have your primary key partitioned in your Kafka? In order to enable upsert, Kafka topic must be partitioned
  • l
    yes
  • v
    Hey everyone, i'm trying to setup pinot from tarbal distribution (so without docker) with an ansible playbook (hopefully will be able to open source it at some point). However i hit a wall when trying to load plugins, i'm using java 8 (
    openjdk version "1.8.0_275"
    ) with the following jvm flags:
    Copy code
    JVM_OPTS=-Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -Dplugins.include=pinot-pubsub,pinot-s3 -Xloggc:/var/log/pinot-gc-controller.log -Dplugins.dir=/usr/local/pinot/plugins
  • v
    The directories are the same as in the docker image:
  • v
    I've a test setup in docker working fine (with the same controller config), however in bare metal with the tar distrib i'm getting:
  • v
    Copy code
    Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.pinot.plugin.filesystem.S3PinotFS
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:58) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:74) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b84>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.controller.ControllerStarter.initPinotFSFactory(ControllerStarter.java:481) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0->
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.controller.ControllerStarter.setUpPinotController(ControllerStarter.java:329) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:287) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:116) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:91) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb6>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:234) ~[pinot-al>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.6.0-jar-wit>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:233) [pinot-all-0.6.0-ja>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:183) [pinot-all-0.6.0-jar-with-dependen>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [pinot-all-0.6.0-jar-with-dependencies.jar>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:154) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646bace>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:166) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafc>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]: Caused by: java.lang.ClassNotFoundException: org.apache.pinot.plugin.filesystem.S3PinotFS
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_275]
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_275]
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.spi.plugin.PluginClassLoader.loadClass(PluginClassLoader.java:80) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646bacea>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:268) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafc>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:239) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafc>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:220) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafc>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:53) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd>
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]:         ... 13 more
  • v
    For more info here are part of the init logs which logs env config:
    Copy code
    Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: ZkClient monitor key or type is not provided. Skip monitoring.
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Starting ZkClient event thread.
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]: Terminate ZkClient event thread.
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]: Terminate ZkClient event thread.
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]: Closed zkclient
Dec 30 16:34:21 ubuntu2004.localdomain bash[10587]: Initializing PinotFSFactory
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:java.class.path=/usr/local/pinot/lib/pinot-all-0.6.0-jar-with-dependencies.jar
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:java.io.tmpdir=/tmp
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:java.compiler=<NA>
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:os.name=Linux
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:os.arch=amd64
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:os.version=5.4.0-54-generic
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:user.name=root
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:user.home=/root
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Client environment:user.dir=/usr/local/apache-pinot-incubating-0.6.0-bin
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Initiating client connection, connectString=10.1.0.11:2181 sessionTimeout=30000 watcher=org.apache.helix.manager.zk.client.ZkConnectionManager@71e9ebae
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Opening socket connection to server 10.1.0.11/10.1.0.11:2181. Will not attempt to authenticate using SASL (unknown error)
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Socket connection established to 10.1.0.11/10.1.0.11:2181, initiating session
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Session establishment complete on server 10.1.0.11/10.1.0.11:2181, sessionid = 0x100000024270010, negotiated timeout = 30000
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: zookeeper state changed (SyncConnected)
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: MBean HelixZkClient:Key=10_1_0_11_2181_30000,Type=ZkConnectionManager has been registered.
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: MBean HelixZkClient:Key=10_1_0_11_2181_30000,PATH=Root,Type=ZkConnectionManager has been registered.
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: ZkConnection 10_1_0_11_2181_30000 was created for sharing.
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Sharing ZkConnection 10_1_0_11_2181_30000 to a new SharedZkClient.
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: ZkClient monitor key or type is not provided. Skip monitoring.
Dec 30 16:34:20 ubuntu2004.localdomain bash[10587]: Starting ZkClient event thread.
  • v
    Do anyone have an idea ?
  • m
    Hello, My text index somehow stopped working. it will give results for some search data however not for all. For eg:  following is working 
    select * from mytable where regexp_likg(log, '0D82F520-62C8-9914-14B8-4C2331E54075')
     (edited)  But this one will not 
    select * from mytable where text_match('0D82F520-62C8-9914-14B8-4C2331E54075')
     (edited)  Any pointers how to debug?
  • k
    There was some bug with stop words @Sidd ^^
  • k
    Is it tokenizing- ?
  • m
    How can i find out whether it is tokenizing? it seems like some data are not going into text index
  • k
    Can you try text match without -
  • m
    that also not working , text match with 
    -
    however will work for some
  • k
    select * from mytable where text_match('0D82F520')
  • k
    I see
  • k
    Does this work?
  • m
    no this will not work for that particular data
  • m
    but will work for someother
  • m
    in my testing all these were working before on an old index, or I it could be I just started bit more extensive testing
  • m
    I deleted the entire cluster and recreated again, but still no luck
  • k
    I don’t think that will fix it, we will try a fix quick test and get back... can you file an issue
  • m
    sure will do that now
  • k
    Looks like a bug to me... is this latest version?
  • m
    yes I upgraded to latest because of this
  • k
    Okay...
  • s
    Hi @Matt, the other day this was the query cache issue
  • s
    Which you had accidentally enabled on your text index
1...139140141...166Latest