image.png

<!here> I want to configure partition aware to routing on my Realtime table.The table already exists and below routing config

{
  "tableName": "events",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "eventTimestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "events",
    "replicasPerPartition": "1",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "365",
    "segmentPushType": "APPEND"
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kinesis",
      "stream.kinesis.topic.name": "events-stream",
      "region": "us-east-1",
      "shardIteratorType": "LATEST",
      "stream.kinesis.consumer.type": "lowlevel",
      "stream.kinesis.fetch.timeout.millis": "30000",
      "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
      "realtime.segment.flush.threshold.size": "5000000",
      "realtime.segment.flush.threshold.time": "1d"
    }
  },
  "upsertConfig": {
    "mode": "FULL"
  },
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  },
  "metadata": {
    "customConfigs": {}
  }
}

I created a new config and made the below changes

{
  "tableName": "events",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "eventTimestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "events",
    "replicasPerPartition": "3",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "1826",
    "segmentPushType": "APPEND",
    "replicaGroupStrategyConfig": {
      "partitionColumn": "accountId",
      "numInstancesPerPartition": 1
    }
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kinesis",
      "stream.kinesis.topic.name": "events-stream",
      "region": "us-east-1",
      "shardIteratorType": "LATEST",
      "stream.kinesis.consumer.type": "lowlevel",
      "stream.kinesis.fetch.timeout.millis": "30000",
      "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
      "realtime.segment.flush.threshold.size": "5000000",
      "realtime.segment.flush.threshold.time": "1d"
    },
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "accountId": {
          "functionName": "Modulo",
          "numPartitions": 4
        }
      }
    }
  },
  "upsertConfig": {
    "mode": "FULL"
  },
  "routing": {
    "segmentPrunerTypes": [
      "partition"
    ],
    "instanceSelectorType": "strictReplicaGroup"
  },
  "metadata": {
    "customConfigs": {}
  }
}

I tried to rebalance the cluster with the below endpoint and have passed the params in the image.

/tables/{tableName}/rebalance

Hellos, where can I find documentation on what does

validationTypesToSkip

mean for Pinot

PUT /tables/{tableName}

api :

image.png

Hi @Mayank 👋 I'll be working with @Scott deRegt to have both time based & partition based segment pruning for our Pinot tables. I was reading through this thread and wanted to get some guidance around the

partition function

&

numPartitions

. Based on this doc, I saw that

Pinot currently supports Modulo, Murmur, ByteArray and HashCode hash functions

. Any guidance around when to choose which function & also, your recommendation to use Murmur (not modulo)? Thread in Slack Conversation

And I see the segments in s3 and directly in the pinot UI.