Hello, I am looking to understand a bit more how t...
# general
a
Hello, I am looking to understand a bit more how the combination of
segmentPrunerTypes
works when pruning segments for a query: 🧵
We have a setup where we have a time column, let's say "timestamp", and then we have some other ID columns like "memberId" and "eventTypeId". In our tableIndexConfig we have something like
Copy code
"tableIndexConfig": {
      "segmentPartitionConfig": {
        "columnPartitionMap": {
          "memberId": {
            "functionName": "Murmur",
            "numPartitions": 48
          },
          "eventTypeId": {
            "functionName": "Murmur",
            "numPartitions": 48
          }
        }
      },
and our routing config looks like
Copy code
"routing": {
      "segmentPrunerTypes": [
        "time",
        "partition"
      ],
      "instanceSelectorType": "replicaGroup"
    },
I am curious when we have a query where we are specifying all of these fields in the WHERE clause, like
Copy code
SELECT count(*) count from my_table 
  WHERE memberID = 489
    AND timestamp > 1748964616000
    AND eventTypeId = 4
Is it expected that all of this pruning will work in combination, so the broker can prune segments by these 3 different columns? I have read the docs here which speak to each one individually, but not in combination necessarily https://docs.pinot.apache.org/operators/operating-pinot/tuning/routing
m
Note these partition configs are to notify Pinot that data is partitioned, not to instruct Pinot to actually partition. Having said that if data is already partitioned, the appropriate pruners will kick in
s
is it applicable to only offline table?
m
Which part?
s
"segmentPartitionConfig": { "columnPartitionMap": { "operation_context": { "functionName": "Murmur", "numPartitions": 64 } } },
m
No, it applies to both. But in both cases you are telling Pinot data is already partitioned, not asking it to partition
s
can we put timestamp column
m
No, pruning happens on top of this so no need to specify here