We have a realtime table and increased the # of replicas from 1 to 3. Looks like it registered on the endpoint, how to verify that there are 3 replicas?

Or does a call to rebalance need to be done first?

@Buchi Reddy Based on the log, you need to wait for the segment to become available before querying it

@Elon After changing the # of replicas in your table config, you need to rebalance the table so that it can take effect

The rebalance result should show 3 replicas for each segment

Thanks:) Great! Just submitted it. Says "IN_PROGRESS" - how do I monitor the rebalance progress, just resubmit the same call?

@Elon You need to check the controller log for the progress

Or wait for a while and resubmit the call to verify that it is done, where you should see

NO_OP

Thanks!

One last question - should we enable "include consuming" - looks like each online segment has 3 replicas

nvm - I included consuming and reissued - the rebalance succeeded. Thanks @Jackie!!!

Hello, I am trying to run two similar SQL statements that differ only by column order: 1. SELECT colA, colB, colC, SUM(metricA) from table GROUP BY colA, colB, colC 2. SELECT colC, colA, colB, SUM(metricA) from table GROUP BY colC, colA, colB SQL #2 returns results. But SQL #1 ends up in exception:

"exceptions": [
    {
        "message": "QueryExecutionError:\njava.lang.RuntimeException: Caught exception while building data table.\n\tat org.apache.pinot.core.operator.blocks.InstanceResponseBlock.<init>(InstanceResponseBlock.java:46)\n\tat org.apache.pinot.core.operator.InstanceResponseOperator.getNextBlock(InstanceResponseOperator.java:37)\n\tat org.apache.pinot.core.operator.InstanceResponseOperator.getNextBlock(InstanceResponseOperator.java:26)\n\tat org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49)\n\tat org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:48)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:220)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:152)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:136)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)\n\tat shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)",
        "errorCode": 200
    }],

Any ideas?

yes, @Neha Pawar found it working too.

Just an fyi we tried replica group and column wise partitioning , reduced pinot servers from 4 to 2, each of 7 cores. With load of 5k ingestion and 7.5k querying for an hour of PT, we are getting 95th percentile from pinot servers as 9 ms and brokers around 12 ms

Nice!

Total data volume during this time was 11 million and it eventually grew upto 20 million

And around 20 to 30 segments..

How big is each segment? And total data size in GB?

Both these techniques greatly improved the response time

20 mb

Hmm, is this real-time only?

Yes relatime

Ok, you can get more improvement by increasing segment size

Throughput

By increasing segment flush threshold

Hmm as per the utility the clt the recommended size was 50 mb but v reduced it to 20 mb , because of the time each segment is taking to get created for our tests. V will retry with 50 mb

Which utility?

Typically we recommend few hundred MBs to 1-2 GB

Realtime procisioning helper

Are you looking to improve performance further ?