numSegmentsQueried=21380, numSegmentsProcessed=90

Hi, we haven’t set any bloom filters and don’t use any partitioning. The query has a time range filter. I just wanted to confirm that in the above example, time-based pruning of segments happened at the broker level and after the broker layer, only 90 segments were queried in the server, right? And if we set bloom filters, are the segments pruned at the server or broker?

i’m not sure about bloom filters doing pruning at a broker level but this for sure would do pruning at the broker level https://docs.pinot.apache.org/operators/operating-pinot/tuning/routing#partitioning

If you’ve not set routingConfig in table config, afaik there will be no broker side pruning. The pruning you’re seeing is happening on server side, via the ColumnValueSegmentPruner, which prunes based on the RANGE or EQ predicates. if you click on

show JSON format

in the result, you should see

"numSegmentsPrunedByBroker": 0,
  "numSegmentsPrunedByServer": 6,

to confirm

is that a feature in newer pinot? i’m on

0.10.0

and i don’t see that

@Neha Pawar I thought the time-based pruning happens during the routing table calculation at the broker? So this means that the broker is querying all of these segments, and due to the server-side pruning is processed only 90 segments?

So for adding broker level time pruning, Is below the required configuration?

"routing": {
    "segmentPrunerTypes": ["time"]
  }

yes, the metadata side metrics does looks pretty new, committed last month

It seems like we can have more than one time there, could we have [time, partitioning]

yes you could. i’m realizing the documentation is quite little for this as we’re discussing more.. @Mark Needham do you think we could add a recipe for broker side pruning? ^^

yea that bit was a little confused so if we don't set up partitioning there with time then we wouldn't get broker side pruning w time

time and partition are separate pruners. time pruner prunes based on the table’s time column, using start/end time values from segment zk metadata

/**
 * The {@code TimeSegmentPruner} prunes segments based on their time column start & end time metadata stored in ZK.
 * The pruner
 * supports queries with filter (or nested filter) of EQUALITY and RANGE predicates.
 */
public class TimeSegmentPruner implements SegmentPruner {

you dont need to have setup any partition column or partitioning config for this one

This helps. Thanks a lot @Neha Pawar

Hi @Neha Pawar 👋 Reg. this message - we are planning to do both time based & partition based broker side pruning for our Pinot offline table(s). Are there any references and/or a recipe for the same? Other related thread with discussions.

@Neha Pawar do you have to add both pruners (time and partition) if you want to do both pruners?

yes, both will have to be added

if I don't have this setup what would happen, time pruning will happen by default but not partitioning pruning (?)

server side pruning happens automatically wherever applicable. broker side pruning will not happen at all, unless config specified

I see, thank you!

hmm, good question. i want to say neither. but brokers may need a restart to pick up the config. Let me check

hey Neha did you happen to have a chance to check? also how could I check that this is happening

yea so it wont happen automatically. either broker restart, or invoke this API on broker

@PUT
  @Produces(MediaType.TEXT_PLAIN)
  @Path("/routing/{tableName}")
  @ApiOperation(value = "Build/rebuild the routing for a table", notes = "Build/rebuild the routing for a table")
  @ApiResponses(value = {
      @ApiResponse(code = 200, message = "Success"),
      @ApiResponse(code = 500, message = "Internal server error")
  })