https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
h

harnoor

06/16/2022, 12:48 PM
Copy code
numSegmentsQueried=21380, numSegmentsProcessed=90
Hi, we haven’t set any bloom filters and don’t use any partitioning. The query has a time range filter. I just wanted to confirm that in the above example, time-based pruning of segments happened at the broker level and after the broker layer, only 90 segments were queried in the server, right? And if we set bloom filters, are the segments pruned at the server or broker?
l

Luis Fernandez

06/16/2022, 2:16 PM
i’m not sure about bloom filters doing pruning at a broker level but this for sure would do pruning at the broker level https://docs.pinot.apache.org/operators/operating-pinot/tuning/routing#partitioning
n

Neha Pawar

06/16/2022, 7:36 PM
If you’ve not set routingConfig in table config, afaik there will be no broker side pruning. The pruning you’re seeing is happening on server side, via the ColumnValueSegmentPruner, which prunes based on the RANGE or EQ predicates. if you click on
show JSON format
in the result, you should see
Copy code
"numSegmentsPrunedByBroker": 0,
  "numSegmentsPrunedByServer": 6,
to confirm
thankyou 1
l

Luis Fernandez

06/16/2022, 8:39 PM
is that a feature in newer pinot? i’m on
0.10.0
and i don’t see that
1
p

Prashant Pandey

06/17/2022, 8:13 AM
@Neha Pawar I thought the time-based pruning happens during the routing table calculation at the broker? So this means that the broker is querying all of these segments, and due to the server-side pruning is processed only 90 segments?
h

harnoor

06/17/2022, 11:34 AM
So for adding broker level time pruning, Is below the required configuration?
Copy code
"routing": {
    "segmentPrunerTypes": ["time"]
  }
n

Neha Pawar

06/17/2022, 8:30 PM
yes, the metadata side metrics does looks pretty new, committed last month
@Prashant Pandey time-based does happen at broker, only if explicitly enabled, as I’m seeing in the code
Copy code
public static List<SegmentPruner> getSegmentPruners(TableConfig tableConfig,
      ZkHelixPropertyStore<ZNRecord> propertyStore) {
    List<SegmentPruner> segmentPruners = new ArrayList<>();
    boolean needsEmptySegment = TableConfigUtils.needsEmptySegmentPruner(tableConfig);
    if (needsEmptySegment) {
      // Add EmptySegmentPruner if needed
      segmentPruners.add(new EmptySegmentPruner(tableConfig, propertyStore));
    }

    RoutingConfig routingConfig = tableConfig.getRoutingConfig();
    if (routingConfig != null) {
      List<String> segmentPrunerTypes = routingConfig.getSegmentPrunerTypes();
      if (segmentPrunerTypes != null) {
But the server side SegmentPruner will always kick in .
@harnoor - yes about the config
l

Luis Fernandez

06/17/2022, 8:34 PM
It seems like we can have more than one time there, could we have [time, partitioning]
as in more than one segmentPrunerTypes
n

Neha Pawar

06/17/2022, 8:45 PM
yes you could. i’m realizing the documentation is quite little for this as we’re discussing more.. @Mark Needham do you think we could add a recipe for broker side pruning? ^^
👀 1
l

Luis Fernandez

06/17/2022, 9:03 PM
yea that bit was a little confused so if we don't set up partitioning there with time then we wouldn't get broker side pruning w time
n

Neha Pawar

06/18/2022, 12:43 AM
time and partition are separate pruners. time pruner prunes based on the table’s time column, using start/end time values from segment zk metadata
Copy code
/**
 * The {@code TimeSegmentPruner} prunes segments based on their time column start & end time metadata stored in ZK.
 * The pruner
 * supports queries with filter (or nested filter) of EQUALITY and RANGE predicates.
 */
public class TimeSegmentPruner implements SegmentPruner {
you dont need to have setup any partition column or partitioning config for this one
h

harnoor

06/20/2022, 7:15 AM
This helps. Thanks a lot @Neha Pawar
n

Neeraja Sridharan

08/10/2022, 7:26 PM
Hi @Neha Pawar 👋 Reg. this message - we are planning to do both time based & partition based broker side pruning for our Pinot offline table(s). Are there any references and/or a recipe for the same? Other related thread with discussions.
l

Luis Fernandez

08/17/2022, 9:37 PM
@Neha Pawar do you have to add both pruners (time and partition) if you want to do both pruners?
n

Neha Pawar

08/18/2022, 3:51 PM
yes, both will have to be added
l

Luis Fernandez

08/18/2022, 3:52 PM
if I don't have this setup what would happen, time pruning will happen by default but not partitioning pruning (?)
n

Neha Pawar

08/18/2022, 3:54 PM
server side pruning happens automatically wherever applicable. broker side pruning will not happen at all, unless config specified
l

Luis Fernandez

08/18/2022, 3:55 PM
I see, thank you!
do I need to run a rebalance if I were to add this? Or just segment reload
n

Neha Pawar

08/18/2022, 3:58 PM
hmm, good question. i want to say neither. but brokers may need a restart to pick up the config. Let me check
l

Luis Fernandez

08/18/2022, 7:04 PM
hey Neha did you happen to have a chance to check? also how could I check that this is happening
Copy code
"routing": {
    "segmentPrunerTypes": [
      "partition"
    ]
  },
this is our current config
does that mean that only partition pruning is happening but not time (?)
in the broker that is
n

Neha Pawar

08/18/2022, 8:18 PM
yea so it wont happen automatically. either broker restart, or invoke this API on broker
Copy code
@PUT
  @Produces(MediaType.TEXT_PLAIN)
  @Path("/routing/{tableName}")
  @ApiOperation(value = "Build/rebuild the routing for a table", notes = "Build/rebuild the routing for a table")
  @ApiResponses(value = {
      @ApiResponse(code = 200, message = "Success"),
      @ApiResponse(code = 500, message = "Internal server error")
  })
correct about
does that mean that only partition pruning is happening but not time (?)