troywinter
05/10/2021, 3:56 AMRK
05/10/2021, 5:16 AMRK
05/10/2021, 5:17 AMSyed Akram
05/10/2021, 7:18 AMPedro Silva
05/10/2021, 9:17 AMPedro Silva
05/10/2021, 2:53 PMtroywinter
05/11/2021, 3:43 AMPedro Silva
05/11/2021, 10:08 AMAmbika
05/11/2021, 2:58 PMPedro Silva
05/11/2021, 3:41 PMSanthi Kollipara
05/11/2021, 4:53 PMVengatesh Babu
05/11/2021, 5:15 PMAmbika
05/12/2021, 2:32 AMAmbika
05/12/2021, 8:41 AMtroywinter
05/12/2021, 12:06 PM[
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.IndexOutOfBoundsException\n\tat java.nio.Buffer.checkBounds(Buffer.java:571)\n\tat java.nio.DirectByteBuffer.get(DirectByteBuffer.java:264)\n\tat org.apache.pinot.core.segment.index.readers.forward.VarByteChunkSVForwardIndexReader.getStringCompressed(VarByteChunkSVForwardIndexReader.java:80)\n\tat org.apache.pinot.core.segment.index.readers.forward.VarByteChunkSVForwardIndexReader.getString(VarByteChunkSVForwardIndexReader.java:60)\n\tat org.apache.pinot.core.segment.index.readers.forward.VarByteChunkSVForwardIndexReader.getString(VarByteChunkSVForwardIndexReader.java:34)\n\tat org.apache.pinot.core.common.DataFetcher$ColumnValueReader.readStringValues(DataFetcher.java:465)\n\tat org.apache.pinot.core.common.DataFetcher.fetchStringValues(DataFetcher.java:146)\n\tat org.apache.pinot.core.common.DataBlockCache.getStringValuesForSVColumn(DataBlockCache.java:194)\n\tat org.apache.pinot.core.operator.docvalsets.ProjectionBlockValSet.getStringValuesSV(ProjectionBlockValSet.java:94)\n\tat org.apache.pinot.core.common.RowBasedBlockValueFetcher.createFetcher(RowBasedBlockValueFetcher.java:64)\n\tat org.apache.pinot.core.common.RowBasedBlockValueFetcher.<init>(RowBasedBlockValueFetcher.java:32)\n\tat org.apache.pinot.core.operator.query.SelectionOrderByOperator.computePartiallyOrdered(SelectionOrderByOperator.java:237)\n\tat org.apache.pinot.core.operator.query.SelectionOrderByOperator.getNextBlock(SelectionOrderByOperator.java:178)\n\tat org.apache.pinot.core.operator.query.SelectionOrderByOperator.getNextBlock(SelectionOrderByOperator.java:73)"
}
]
Ricardo Bernardino
05/12/2021, 1:29 PMRK
05/12/2021, 5:04 PMAmbika
05/12/2021, 5:36 PMVengatesh Babu
05/12/2021, 6:50 PMAaron Wishnick
05/12/2021, 7:07 PMAkash
05/12/2021, 7:48 PM"tableIndexConfig" : {
"starTreeIndexConfigs":[{
"maxLeafRecords": 1000,
"functionColumnPairs": ["DISTINCT_COUNT_HLL__user_id","COUNT__dt"],
"dimensionsSplitOrder": ["dt","dim1","dim2","dim3","dim4"]
}],
"enableDynamicStarTreeCreation" : true
},
This is to optimise following queries.
select dt,DISTINCT_COUNT_HLL(user_id) FROM TABLE GROUP BY dt
select dt,count(1) FROM TABLE GROUP BY dt
select dt,dim2,DISTINCT_COUNT_HLL(user_id) FROM TABLE where dim1 = 3 GROUP BY dt, dim2
select dt,dim2,count(1) FROM TABLE where dim1 = 3 GROUP BY dt, dim2
dim1,2,3,4 does not have too much high cardinality. User_id has the biggest cardinality.Yupeng Fu
05/12/2021, 10:33 PMtroywinter
05/13/2021, 3:12 AMVengatesh Babu
05/13/2021, 12:08 PMcolumn.RELATEDID.partitionFunction = Murmur
column.RELATEDID.numPartitions = 10
column.RELATEDID.partitionValues = 0,1,2,3,4,5,6,7,8,9
Note: Running Data Ingestion using pinot-admin.sh LaunchDataIngestionJobtroywinter
05/13/2021, 2:22 PMPedro Silva
05/13/2021, 3:24 PMPedro Silva
05/13/2021, 4:50 PMArun Vasudevan
05/13/2021, 6:13 PMReload All Segments
for the Table in the UI.
2 Questions here:
• I see Reload All Segments
to re-index data is this the right approach to re-populate new columns?
• I don’t see the progress of the Reload All Segments
I see this PR is completed - https://github.com/apache/incubator-pinot/issues/5390 which release is this part of?Aaron Wishnick
05/14/2021, 5:28 PMselect foo, percentiletdigest(bar, 0.5) from mytable group by foo
. I've got foo
in my dimensionsSplitOrder
and I've got PERCENTILE_TDIGEST__bar
as well as AVG__bar
in my functionColumnPairs
. My query takes about 700 ms but if I switch it to avg(bar)
it takes 15 ms. Is it expected that the t-digest would be that much slower? Anything I can do to speed it up?Vishnu
05/16/2021, 2:14 PM