https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • x
    hello, after following the steps in to load my data here https://docs.pinot.apache.org/integrations/presto i get this error:
    Copy code
    Query 20210731_123555_00024_7fd9g failed: Query SELECT count(*) FROM events WHERE (time > 1622085300) encountered exception {"message":"PQLParsingError:\norg.apache.pinot.sql.parsers.SqlCompilationException: Caught exception while parsing query: SELECT count(*) FROM events WHERE (time > 1622085300)\n\tat org.apache.pinot.sql.parsers.CalciteSqlParser.compileCalciteSqlToPinotQuery(CalciteSqlParser.java:324)\n\tat org.apache.pinot.sql.parsers.CalciteSqlParser.compileToPinotQuery(CalciteSqlParser.java:108)\n\tat org.apache.pinot.sql.parsers.CalciteSqlCompiler.compileToBrokerRequest(CalciteSqlCompiler.java:35)\n\tat org.apache.pinot.core.requesthandler.PinotQueryParserFactory.parseSQLQuery(PinotQueryParserFactory.java:46)\n\tat org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleSQLRequest(BaseBrokerRequestHandler.java:212)\n\tat org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:194)\n\tat org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:99)\n\tat org.apache.pinot.broker.api.resources.PinotClientRequest.processSqlQueryPost(PinotClientRequest.java:175)\n\tat jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)","errorCode":150} with pinot query "SELECT count(*) FROM events WHERE (time > 1622085300)"
    in pinot, time schema is like such:
    Copy code
    "dateTimeFieldSpecs": [{
     "name": "time",
     "dataType": "LONG",
     "format" : "1:SECONDS:EPOCH",
     "granularity": "15:MINUTES"
   }]
    and when i run the supposedly erroneous query in pinot
    Copy code
    SELECT count(*) FROM events WHERE (time > 1622085300)
    there are no issues.
    k
    • 2
    • 4
  • x
    i’m trying to run the presto coordinator for 
    pinot-quickstart
    with extra configuration (https://docs.pinot.apache.org/integrations/presto#using-pinot-streaming-grpc-connector)
    Copy code
    docker run   \
--network pinot-demo   \
--name=presto-coordinator   \
-p 8080:8080   \
--mount type=bind,source=$SOURCE_DIR,target=/home/presto/data -d \
apachepinot/pinot-presto:latest
    but it doesn’t seem like i’m allowed to do so? https://github.com/apache/pinot/blob/master/docker/images/pinot-presto/Dockerfile#L55 i’m not very proficient with docker, can someone suggest how i can override the volume?
    k
    x
    • 3
    • 7
  • b
    Q on consuming from kafka - if the table is created through the UI is there some endpoint that must be hit to start the consumer? As far as I can tell I've created the table correctly, it validated in the UI and said it could connect to kafka, but it just doesn't seem to consume any messages. I've verified that I can connect to kafka from the controller/broker machine and there there is messages in the topic. The broker and controller logs do not seem to contain anything related to kafka however
    m
    • 2
    • 9
  • b
    Is there any reason why a realtime table would create a segment that immediately is shown as bad?
    m
    • 2
    • 7
  • j
  • s
    Hi @Mayank @Xiang Fu the table query response becomes unstable on pinot sql query editor once we updated the schema of the table means some times it returns rows in response .. somtimes rows are empty .. the query does group by , select columns and filter cc: @Mohamed Sultan @Mohamed Kashifuddin @Manju Priyadharshini
    x
    m
    +2
    • 5
    • 44
  • p
    Any idea how to find the kafka serialization exception which is breaking my new real time confluent avro kafka flow. Im not finding the right log message/full stack trace. I can see the controller is getting sent a segment exception update from I guess the server process.
    m
    • 2
    • 6
  • n
    Hey all, we have several newly added real-time tables are facing a problem. Their segments never turn to done. Do you have any idea how to fix that?
    m
    • 2
    • 8
  • r
    🧵Backfill times out- 
    <http://shaded.org|shaded.org>.apache.http.NoHttpResponseException: external_controller_uri:9000 failed to respond
    . I've done a backfill job on a new _OFFLINE table twice and in the last part of it, I've twice had a HTTP response error about 7 hours into a job.
    m
    k
    • 3
    • 12
  • d
    It is giving same result each time when it is trying to schedule task in a periodic way
    j
    • 2
    • 2
  • w
    Hi, I am trying to move a table to a different set of servers by performing a rebalance, but I don't think my segments are moving. The table itself is listed under the new servers but not the segments. After launching the rebalance, I get a message "In progress, check controller logs for updates", but the controller logs don't have anything.
    m
    • 2
    • 2
  • a
    even after adding this parameter in server config - pinot.server.instance.reload.consumingSegment=true result remains the same.
    j
    • 2
    • 2
  • k
    Hello everyone, Server Scale In Testing work in progress Invalid command caused the Server Status of Segment to be Offline. How can I change this state to Online? Work progress 1. The state of server-10 and server-11 out of 12 servers was made dead. 2. curl -X POST "https://xxxxxx.com:9443/segments/xx_config_OFFLINE/reset?maxWaitTimeMs=30000" -H "accept: application / json" And The response to the TableRebalancer command "Cannot reassign OFFLINE instances (instance assignment is not allowed) for table: xx_OFFLINE" What's wrong with this?
    j
    • 2
    • 7
  • s
    does pinot broker query call happens with only http or ssl can also be enabled .. @Mayank @Xiang Fu @Jackie cc: @Mohamed Kashifuddin @Shailesh Jha
    m
    • 2
    • 2
  • e
    Hello! I’m trying to use the latest 0.8.0 rc with the Pulsar connector (apachepinot/pinot:0.8.0-SNAPSHOT-573651b28-20210729-jdk8 or apachepinot/pinot:0.8.0-SNAPSHOT-573651b28-20210729-jdk11 docker image). But I’m having this error when creating a table:
    Copy code
    pinot-controller_1  | 2021/08/03 12:00:19.766 ERROR [PulsarPartitionLevelConnectionHandler] [grizzly-http-server-3] Could not create pulsar consumer
pinot-controller_1  | org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: 'java.lang.String org.apache.commons.codec.digest.DigestUtils.sha1Hex(java.lang.String)'
pinot-controller_1  | 	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:978) ~[pinot-pulsar-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-573651b28a6f89bd4895c992a5e8fa8e23df4615]
pinot-controller_1  | 	at org.apache.pulsar.client.impl.ReaderBuilderImpl.create(ReaderBuilderImpl.java:77) ~[pinot-pulsar-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-573651b28a6f89bd4895c992a5e8fa8e23df4615]
pinot-controller_1  | 	at org.apache.pinot.plugin.stream.pulsar.PulsarPartitionLevelConnectionHandler.<init>(PulsarPartitionLevelConnectionHandler.java:56) ~[pinot-pulsar-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-573651b28a6f89bd4895c992a5e8fa8e23df4615]
pinot-controller_1  | 	at org.apache.pinot.plugin.stream.pulsar.PulsarStreamMetadataProvider.<init>(PulsarStreamMetadataProvider.java:51) ~[pinot-pulsar-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-573651b28a6f89bd4895c992a5e8fa8e23df4615]
    Do you have any idea why this is happening? Thank you
    k
    s
    • 3
    • 4
  • b
    I've noticed that stream ingestion with kafka/json the fields are case sensitive and have to match exactly between the json and pinot config. Is there an option somewhere that I've missed to make that case insensitive?
    j
    • 2
    • 1
  • l
    Hi All, we are using 0.7.1 version with gcs as the deepstore. In a perf test, we are seeing controller becoming the bottleneck in segment upload path to gcs. And we came across the following doc to overcome the controller bottleneck https://docs.pinot.apache.org/operators/operating-pinot/decoupling-controller-from-the-data-path Have a basic question here? • How split commit is related to peer download? As per my understanding, commit is in the ingestion path and needs upload to deep(gcs). And from peer download feature, I understand that segments are downloaded from other peer servers in case of deepstore(gcs) unavailability. Though, I had gone through this code path and figured out the implementation is same as documentation, I didn’t understand how and why these two (upload while committing and peer download) are coupled?
    k
    s
    • 3
    • 40
  • c
    We have an offline Pinot table with 20 columns experiencing high latency or timeout when query it with large offset: e.g. 1000000. some of the columns has large text: e.g. 50000 chars. Is there some thing we could change to help improving the query latency?
    m
    x
    k
    • 4
    • 17
  • n
    Hi, I have a use case for REALTIME table with updates with kafka as streaming source. At the same time, I have to pull in existing data from s3 to pinot as well for same use case so that the complete data can be queries under one table. Is there any way to achieve this?
    k
    • 2
    • 4
  • d
    Hello ! , i am working on managing offline flows ( from realtime to offline table) with bucketTimePeriod - 4h with 2M record with bufferTimePeriod - 5d which will move data into offline table daily with maxNumRecordsPerSegment-10M .How can we manage memory in realtime table with 5 days bufferTimePeriod?
    m
    • 2
    • 4
  • a
    Hey guys, I having trouble connecting to pinot through jdbc, I tried from jmeter and DataGrip from jmeter I get 
    java.lang.NoClassDefFoundError: org/apache/pinot/client/JsonAsyncHttpPinotClientTransportFactory
    and from DataGrip, when I try to add a datasource I get 
    Driver class 'org.slf4j.LoggerFactory' not found
    if anyone has a working jdbc configuration, could you please share
    k
    k
    • 3
    • 6
  • b
    can we update dimension's default value in pinot ?
    m
    j
    • 3
    • 6
  • c
    Hi team, we are have a latency issue with a Pinot query for UI, it’s simple doing a select sum of a Int column from a table with 2 billion total records with 6milliom records scanned after filtering by one inverted index and timecolumnname, but it always took more than 2 seconds. Is this normal and can this latency be improved?
    j
    k
    k
    • 4
    • 7
  • a
    Hello Everyone, I am trying to move data from REALTIME to OFFLINE table after schema evolution. I have added a new column age of type INT and since I did not provide any value it is reflecting 0 in the real-time table , but when the task RealtimeToOfflineSegmentsTask is getting executed in minion console it is throwing exception "INFO": "org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: null of int in field age of record" if anyone has any idea regarding the same , please do share
    m
    j
    x
    • 4
    • 6
  • s
    "ingestionConfig": { "transformConfigs": [{ "columnName": "visits_case", "transformFunction": "Groovy((CASE WHEN visits = 1 and (first_tab = 'true' or first_tab = 'null') then 1 else 0 end), visits_case)" } }
    n
    • 2
    • 15
  • r
    Hi!! I’m experiencing some issues with Schema evolution. I have a REALTIME table consuming from a kafka topic with a defined schema. if I add a new column to the table initially the column doesn’t appear but after reloading the segments the table appears in queries but with 
    null
    value. The only way I have achieved to see data on the new column is restarting the server instances. Any idea about I have to do to get the data n the table after update the schema? Note: following the documentation I added the property: 
    pinot.server.instance.reload.consumingSegment=true
    to my 
    pinot-server-config
    but the result is the same.
    m
    j
    +2
    • 5
    • 44
  • m
    Hi we are querying Pinot via Trino with dynamic tables and have recently encountered this error:
    Copy code
    SELECT * FROM pinot.default."SELECT max(COL1) from table0";
Query 20210810_205808_00168_gj79s failed: class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap')
    COL1
    can be type of INT, LONG, FLOAT, or DOUBLE in here...not sure if someone has seen this before?
    j
    e
    a
    • 4
    • 7
  • a
    Hi team, I have been able to deploy apache pinot with latest master from git , I am able to run the default cluster with command bin/quick-start-batch.sh but when I am trying to run my own cluster, I am able to start the zookeeper , but am getting error while running below service manager command: bin/pinot-admin.sh StartServiceManager  -zkAddress localhost:2181 -clusterName pinot-quickstart  -port -1  -bootstrapConfigPaths  ${PINOT_DIR}/config/pinot-controller.conf  ${PINOT_DIR}/config/pinot-broker.conf ${PINOT_DIR}/config/pinot-server.conf The error is: Error: option "-clusterName" cannot be used with the option(s) [-bootstrapConfigPaths, -bootstrapServices] Although I have been able to run my own cluster previously, but it is showing error with latest master. any idea regarding the same?
    x
    • 2
    • 1
  • x
    Copy code
    $ time python3 queries.py
found 6812 userids

$ time python3 queries.py
found 6782 userids

$ time python3 queries.py
found 6895 userids
    m
    • 2
    • 20
  • d
    @Mayank , i am working on manage offline flows and pushed 20k record with this table config with “bucketTimePeriod”: “1h” and “bufferTimePeriod”: “4h” . ingesting 1 hour late data into realtime segment with 10k records. It gives error like
    transcript-table-realtime.json
    m
    n
    • 3
    • 3
1...192021...166Latest