https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • e

    Elon

    05/21/2020, 6:03 PM
    We have a realtime table and increased the # of replicas from 1 to 3. Looks like it registered on the endpoint, how to verify that there are 3 replicas?
  • e

    Elon

    05/21/2020, 6:04 PM
    Or does a call to rebalance need to be done first?
  • j

    Jackie

    05/21/2020, 6:04 PM
    @Buchi Reddy Based on the log, you need to wait for the segment to become available before querying it
  • j

    Jackie

    05/21/2020, 6:07 PM
    @Elon After changing the # of replicas in your table config, you need to rebalance the table so that it can take effect
    👍 1
  • j

    Jackie

    05/21/2020, 6:08 PM
    The rebalance result should show 3 replicas for each segment
  • e

    Elon

    05/21/2020, 6:09 PM
    Thanks:) Great! Just submitted it. Says "IN_PROGRESS" - how do I monitor the rebalance progress, just resubmit the same call?
  • j

    Jackie

    05/21/2020, 6:11 PM
    @Elon You need to check the controller log for the progress
  • j

    Jackie

    05/21/2020, 6:12 PM
    Or wait for a while and resubmit the call to verify that it is done, where you should see
    NO_OP
  • e

    Elon

    05/21/2020, 6:15 PM
    Thanks!
  • e

    Elon

    05/21/2020, 6:17 PM
    One last question - should we enable "include consuming" - looks like each online segment has 3 replicas
  • e

    Elon

    05/21/2020, 6:25 PM
    nvm - I included consuming and reissued - the rebalance succeeded. Thanks @Jackie!!!
    👍 1
  • k

    kish

    05/22/2020, 8:08 PM
    Hello, I am trying to run two similar SQL statements that differ only by column order: 1. SELECT colA, colB, colC, SUM(metricA) from table GROUP BY colA, colB, colC 2. SELECT colC, colA, colB, SUM(metricA) from table GROUP BY colC, colA, colB SQL #2 returns results. But SQL #1 ends up in exception:
    Copy code
    "exceptions": [
        {
            "message": "QueryExecutionError:\njava.lang.RuntimeException: Caught exception while building data table.\n\tat org.apache.pinot.core.operator.blocks.InstanceResponseBlock.<init>(InstanceResponseBlock.java:46)\n\tat org.apache.pinot.core.operator.InstanceResponseOperator.getNextBlock(InstanceResponseOperator.java:37)\n\tat org.apache.pinot.core.operator.InstanceResponseOperator.getNextBlock(InstanceResponseOperator.java:26)\n\tat org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49)\n\tat org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:48)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:220)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:152)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:136)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)\n\tat shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)",
            "errorCode": 200
        }],
    Any ideas?
  • k

    kish

    05/22/2020, 8:22 PM
    yes, @Neha Pawar found it working too.
  • s

    srisudha

    05/23/2020, 1:29 AM
    Just an fyi we tried replica group and column wise partitioning , reduced pinot servers from 4 to 2, each of 7 cores. With load of 5k ingestion and 7.5k querying for an hour of PT, we are getting 95th percentile from pinot servers as 9 ms and brokers around 12 ms
  • m

    Mayank

    05/23/2020, 1:30 AM
    Nice!
  • s

    srisudha

    05/23/2020, 1:30 AM
    Total data volume during this time was 11 million and it eventually grew upto 20 million
  • s

    srisudha

    05/23/2020, 1:31 AM
    And around 20 to 30 segments..
  • m

    Mayank

    05/23/2020, 1:31 AM
    How big is each segment? And total data size in GB?
  • s

    srisudha

    05/23/2020, 1:31 AM
    Both these techniques greatly improved the response time
  • s

    srisudha

    05/23/2020, 1:31 AM
    20 mb
  • m

    Mayank

    05/23/2020, 1:31 AM
    Hmm, is this real-time only?
  • s

    srisudha

    05/23/2020, 1:31 AM
    Yes relatime
  • m

    Mayank

    05/23/2020, 1:32 AM
    Ok, you can get more improvement by increasing segment size
  • m

    Mayank

    05/23/2020, 1:32 AM
    Throughput
  • m

    Mayank

    05/23/2020, 1:32 AM
    By increasing segment flush threshold
  • s

    srisudha

    05/23/2020, 1:33 AM
    Hmm as per the utility the clt the recommended size was 50 mb but v reduced it to 20 mb , because of the time each segment is taking to get created for our tests. V will retry with 50 mb
  • m

    Mayank

    05/23/2020, 1:34 AM
    Which utility?
  • m

    Mayank

    05/23/2020, 1:35 AM
    Typically we recommend few hundred MBs to 1-2 GB
  • s

    srisudha

    05/23/2020, 1:35 AM
    Realtime procisioning helper
  • m

    Mayank

    05/23/2020, 1:35 AM
    Are you looking to improve performance further ?
1...105106107...166Latest