Hi I am experimenting with instance assignment for partition Apache Pinot #troubleshooting

Hi, I am experimenting with instance assignment fo...

Aditya

01/19/2022, 11:38 AM

Hi, I am experimenting with instance assignment for partitions Trying to get all the segments for a partition in one server, with no replication (1 replica)

Copy code

"instanceAssignmentConfigMap": {
      "OFFLINE": {
        "tagPoolConfig": {
          "tag": "DefaultTenant_OFFLINE",
          "poolBased": false,
          "numPools": 0
        },
        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 2,
          "numReplicaGroups": 1,
          "numInstancesPerReplicaGroup": 2,
          "numPartitions": 4,
          "numInstancesPerPartition": 1
        }
      }
    },

Getting this error when trying rebalance server dry run

Copy code

"description": "Caught exception while calculating target assignment: java.lang.IllegalStateException: Instance partitions: transactions_partitioned_OFFLINE should contain 1 replica and 1 partition for non-replica-group based assignment"

What could be wrong with this config?

Aditya

01/19/2022, 11:42 AM

Complete table config :

Copy code

{
  "OFFLINE": {
    "tableName": "transactions_partitioned_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "schemaName": "transactions_partitioned",
      "replication": "1",
      "segmentPushFrequency": "HOURLY",
      "segmentPushType": "APPEND",
      "timeColumnName": "created_at_epoch",
      "allowNullTimeValue": false
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [],
      "noDictionaryColumns": [],
      "rangeIndexColumns": [],
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [
        "user_id"
      ],
      "bloomFilterColumns": [],
      "loadMode": "MMAP",
      "onHeapDictionaryColumns": [],
      "varLengthDictionaryColumns": [],
      "enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
      "segmentPartitionConfig": {
        "columnPartitionMap": {
          "user_id": {
            "functionName": "Murmur",
            "numPartitions": 4
          }
        }
      },
      "aggregateMetrics": false,
      "nullHandlingEnabled": false
    },
    "metadata": {},
    "quota": {},
    "routing": {
      "segmentPrunerTypes": [
        "partition"
      ],
      "instanceSelectorType": "replicaGroup"
    },
    "query": {},
    "instanceAssignmentConfigMap": {
      "OFFLINE": {
        "tagPoolConfig": {
          "tag": "DefaultTenant_OFFLINE",
          "poolBased": false,
          "numPools": 0
        },
        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 2,
          "numReplicaGroups": 1,
          "numInstancesPerReplicaGroup": 2,
          "numPartitions": 4,
          "numInstancesPerPartition": 1
        }
      }
    },
    "fieldConfigList": [],
    "ingestionConfig": {
      "transformConfigs": [
        {
          "columnName": "created_at_epoch",
          "transformFunction": "fromDateTime(\"created_at\", 'yyyy-MM-dd HH:mm:ss.SSS')"
        }
      ]
    },
    "isDimTable": false
  }
}

Aditya

01/19/2022, 11:47 AM

Do I need to specifially tag servers to

DefaultTenant_OFFLINE

? Where can we define the Helix instance config to tag a server? Indicated here : https://docs.pinot.apache.org/operators/operating-pinot/instance-assignment

Aditya

01/19/2022, 5:59 PM

I am trying to reassign all segments for a partition to same server. Tried another config changing numPartitions to 2 as according to doc numPartitions donot need to be equal to actual no of partitions.

Copy code

"replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 2,
          "numReplicaGroups": 1,
          "numInstancesPerReplicaGroup": 2,
          "numPartitions": 2,
          "numInstancesPerPartition": 1
        }

Getting the same error due to pre-condition check while rebalancing servers

Copy code

Caught exception while calculating target assignment: java.lang.IllegalStateException: Instance partitions: transactions_partitioned_OFFLINE should contain 1 replica and 1 partition for non-replica-group based assignment

Please point out any errors in this config

Mayank

01/20/2022, 5:11 AM

@Aditya, if you just have 2 servers and want to assign all segments to one server and use replication of two, then all you need is to set replication factor of 2.

Mayank

01/20/2022, 5:11 AM

That will force both servers to have all segments.

Mayank

01/20/2022, 5:11 AM

Let me know if I misread your requirement.

Aditya

01/20/2022, 5:14 AM

I am trying to distribute segments

Copy code

Server 1 : partition 0 and 1
Server 2 : partition 2 and 3

What you suggested will do this right?

Copy code

Server 1 : partition 0, 1, 2, 3
Server 2 : partition 0, 1, 2, 3

I thought keeping replication at 1 and numInstancesPerPartition as 1 will achieve this

Aditya

01/20/2022, 5:24 AM

Keep segments for a partition in one server and distributed partitions evenly (round robin) among 2 servers

Mayank

01/20/2022, 10:57 PM

I see. @Seunghyun can you take a look at the error?

Aditya

01/28/2022, 6:14 PM

Hi @Mayank @Seunghyun, Thanks for all the help! I tried experimenting with instance assignment again. One conclusion is partitioned replica group based assignment is only possible for numReplicaGroups > 1 Now I am using 4 servers. Goal is to move all segments for a partition to one server within a replica group. Using numInstancesPerPartition = 1, I expected that within the replica group one single server will handle all segments for a partition (2 partitions here so one server will handle one partiton) Config is :

Copy code

"replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 0,
          "numReplicaGroups": 2,
          "numInstancesPerReplicaGroup": 2,
          "numPartitions": 2,
          "numInstancesPerPartition": 1
        }

Rebalanced servers dry run with reassign instances true. It moves all the segments (irrespective of partition) to just one server per replica group and other server is assigned no segments. The partitions should have been evenly distributed among the 2 servers in a replica group. What is the significance of 1_0 and 1_1 keys in partitionToInstancesMap? It seams they are not used in actual segment assignment

Copy code

{
  "status": "DONE",
  "description": "Dry-run mode",
  "instanceAssignment": {
    "OFFLINE": {
      "instancePartitionsName": "transcript_OFFLINE",
      "partitionToInstancesMap": {
        "0_0": [
          "Server_10.9.9.255_8098"
        ],
        "0_1": [
          "Server_10.9.9.255_9098"
        ],
        "1_0": [
          "Server_10.9.3.206_8098"
        ],
        "1_1": [
          "Server_10.9.5.78_8098"
        ]
      }
    }
  },
  "segmentAssignment": {
    "transcript_OFFLINE_1570863600000_1570863600000_0": {
      "Server_10.9.9.255_8098": "ONLINE",
      "Server_10.9.9.255_9098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_1": {
      "Server_10.9.9.255_8098": "ONLINE",
      "Server_10.9.9.255_9098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_2": {
      "Server_10.9.9.255_8098": "ONLINE",
      "Server_10.9.9.255_9098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_3": {
      "Server_10.9.9.255_8098": "ONLINE",
      "Server_10.9.9.255_9098": "ONLINE"
    }
  }
}

Another thread I found in slack (albeit for realtime upsert table) where numInstancesPerPartition = 1 seems to be assigning all segments to one server https://apache-pinot.slack.com/archives/C011C9JHN7R/p1638900028351800 Please point out error in the config. Shall I select different option while rebalancing servers? tried Reassign Instances and Bootstrap

Mayank

01/28/2022, 6:32 PM

Can you try with

"numInstancesPerPartition": 2

? Also can you confirm if the data is partitioned by inspecting the segment metadata?

Aditya

01/28/2022, 6:39 PM

yes confirmed the metadata, facing similar issue with a larger table (every segment is assigned to one server in replica group) for

"numInstancesPerPartition": 2

, It is distributing segments equally, but now a specific partition can go to any of the two servers.

Copy code

{
  "status": "DONE",
  "description": "Instance reassigned in dry-run mode, table is already balanced",
  "instanceAssignment": {
    "OFFLINE": {
      "instancePartitionsName": "transcript_OFFLINE",
      "partitionToInstancesMap": {
        "0_0": [
          "Server_10.9.3.206_8098",
          "Server_10.9.9.255_8098"
        ],
        "0_1": [
          "Server_10.9.5.78_8098",
          "Server_10.9.9.255_9098"
        ],
        "1_0": [
          "Server_10.9.3.206_8098",
          "Server_10.9.9.255_8098"
        ],
        "1_1": [
          "Server_10.9.5.78_8098",
          "Server_10.9.9.255_9098"
        ]
      }
    }
  },
  "segmentAssignment": {
    "transcript_OFFLINE_1570863600000_1570863600000_0": {
      "Server_10.9.3.206_8098": "ONLINE",
      "Server_10.9.5.78_8098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_1": {
      "Server_10.9.3.206_8098": "ONLINE",
      "Server_10.9.5.78_8098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_2": {
      "Server_10.9.9.255_8098": "ONLINE",
      "Server_10.9.9.255_9098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_3": {
      "Server_10.9.9.255_8098": "ONLINE",
      "Server_10.9.9.255_9098": "ONLINE"
    }
  }
}

Jackie

01/28/2022, 6:50 PM

Can you share the partition id of these 4 segments?

Jackie

01/28/2022, 6:52 PM

Copy code

One conclusion is partitioned replica group based assignment is only possible for numReplicaGroups > 1

This should be fixed. Can you please file a github issue to track this?

Aditya

01/28/2022, 6:54 PM

sure

Aditya

01/28/2022, 6:57 PM

Partition ids :

Copy code

./transcript_OFFLINE_1570863600000_1570863600000_0/v3/metadata.properties:column.studentID.partitionValues = 0
./transcript_OFFLINE_1570863600000_1570863600000_1/v3/metadata.properties:column.studentID.partitionValues = 0
./transcript_OFFLINE_1570863600000_1570863600000_2/v3/metadata.properties:column.studentID.partitionValues = 1
./transcript_OFFLINE_1570863600000_1570863600000_3/v3/metadata.properties:column.studentID.partitionValues = 1

Aditya

01/28/2022, 7:14 PM

@Jackie Created github issue https://github.com/apache/pinot/issues/8090

Jackie

01/28/2022, 8:42 PM

Hmm, why does it partition on

studnetID

instead of

user_id

as configured in the table config?

Aditya

01/29/2022, 3:43 AM

I changed the table to a smaller one. The first table I started with had too many (52) segments, made it hard to reason about This is the new table config

Copy code

{
  "OFFLINE": {
    "tableName": "transcript_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "schemaName": "transcript",
      "replication": "1",
      "timeColumnName": "timestampInEpoch",
      "allowNullTimeValue": false,
      "segmentPushFrequency": "DAILY",
      "segmentPushType": "APPEND",
      "replicasPerPartition": "1"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [],
      "noDictionaryColumns": [],
      "rangeIndexColumns": [],
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [],
      "bloomFilterColumns": [],
      "loadMode": "MMAP",
      "onHeapDictionaryColumns": [],
      "varLengthDictionaryColumns": [],
      "enableDefaultStarTree": false,
      "aggregateMetrics": false,
      "enableDynamicStarTreeCreation": false,
      "segmentPartitionConfig": {
        "columnPartitionMap": {
          "studentID": {
            "functionName": "Modulo",
            "numPartitions": 2
          }
        }
      },
      "nullHandlingEnabled": false
    },
    "metadata": {},
    "quota": {},
    "routing": {
      "segmentPrunerTypes": [
        "partition"
      ],
      "instanceSelectorType": "replicaGroup"
    },
    "query": {},
    "instanceAssignmentConfigMap": {
      "OFFLINE": {
        "tagPoolConfig": {
          "tag": "DefaultTenant_OFFLINE",
          "poolBased": false,
          "numPools": 0
        },
        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 0,
          "numReplicaGroups": 2,
          "numInstancesPerReplicaGroup": 2,
          "numPartitions": 2,
          "numInstancesPerPartition": 1
        }
      }
    },
    "ingestionConfig": {},
    "isDimTable": false
  }
}

Jackie

02/03/2022, 12:04 AM

@Aditya Sorry I missed this thread. I found the reason why the partition info is not applied. In the

segmentsConfig

, you need to add the following config:

Copy code

{
  "replicaGroupStrategyConfig": {
    "partitionColumn": "studentID"
  }
}

Aditya

02/03/2022, 5:19 AM

Thanks Jackie Adding replicaGroupStrategyConfig worked, also had to add

"numInstancesPerPartition": 1

else there was validation error

Copy code

INFO [PinotTableRestletResource] [grizzly-http-server-3] Invalid table config: transcript_OFFLINE with error: Missing required creator property 'numInstancesPerPartition' (index 1)
 at [Source: (String)"{"tableName":"transcript_OFFLINE","tableType":"OFFLINE","segmentsConfig":{"schemaName":"transcript","replication":"1","timeColumnName":"timestampInEpoch","allowNullTimeValue":false,"segmentPushFrequency":"DAILY","segmentPushType":"APPEND","replicasPerPartition":"1","replicaGroupStrategyConfig":{"partitionColumn":"studentID"}},"tenants":{"broker":"DefaultTenant","server":"DefaultTenant"},"tableIndexConfig":{"invertedIndexColumns":[],"noDictionaryColumns":[],"rangeIndexColumns":[],"rangeIndexVersi"[truncated 899 chars]; line: 1, column: 326] (through reference chain: org.apache.pinot.spi.config.table.TableConfig["segmentsConfig"]->org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig["replicaGroupStrategyConfig"]->org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig["numInstancesPerPartition"]) exception: Missing required creator property 'numInstancesPerPartition' (index 1)
 at [Source: (String)"{"tableName":"transcript_OFFLINE","tableType":"OFFLINE","segmentsConfig":{"schemaName":"transcript","replication":"1","timeColumnName":"timestampInEpoch","allowNullTimeValue":false,"segmentPushFrequency":"DAILY","segmentPushType":"APPEND","replicasPerPartition":"1","replicaGroupStrategyConfig":{"partitionColumn":"studentID"}},"tenants":{"broker":"DefaultTenant","server":"DefaultTenant"},"tableIndexConfig":{"invertedIndexColumns":[],"noDictionaryColumns":[],"rangeIndexColumns":[],"rangeIndexVersi"[truncated 899 chars]; line: 1, column: 326] (through reference chain: org.apache.pinot.spi.config.table.TableConfig["segmentsConfig"]->org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig["replicaGroupStrategyConfig"]->org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig["numInstancesPerPartition"])

This config worked:

Copy code

"replicaGroupStrategyConfig": {
        "partitionColumn": "studentID",
        "numInstancesPerPartition": 1
      }

Segment assignment after rebalance servers

Copy code

{
  "status": "DONE",
  "description": "Instance reassigned in dry-run mode, table is already balanced",
  "instanceAssignment": {
    "OFFLINE": {
      "instancePartitionsName": "transcript_OFFLINE",
      "partitionToInstancesMap": {
        "0_0": [
          "Server_10.9.9.255_8098"
        ],
        "0_1": [
          "Server_10.9.9.255_9098"
        ],
        "1_0": [
          "Server_10.9.3.206_8098"
        ],
        "1_1": [
          "Server_10.9.5.78_8098"
        ]
      }
    }
  },
  "segmentAssignment": {
    "transcript_OFFLINE_1570863600000_1570863600000_0": {
      "Server_10.9.9.255_8098": "ONLINE",
      "Server_10.9.9.255_9098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_1": {
      "Server_10.9.9.255_8098": "ONLINE",
      "Server_10.9.9.255_9098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_2": {
      "Server_10.9.3.206_8098": "ONLINE",
      "Server_10.9.5.78_8098": "ONLINE"
    },
    "transcript_OFFLINE_1570863600000_1570863600000_3": {
      "Server_10.9.3.206_8098": "ONLINE",
      "Server_10.9.5.78_8098": "ONLINE"
    }
  }
}

"numInstancesPerPartition": 1 is now present at 2 places

replicaGroupStrategyConfig

and

replicaGroupPartitionConfig

, keeping both same for now Any pointers on how to find undocumented config options?

Jackie

02/03/2022, 6:09 AM

It is a legacy config. I'm working on a fix, and will update the documentation accordingly

Aditya

02/03/2022, 6:12 AM

Thank you!

Jackie

02/03/2022, 7:14 AM

Updated the documentation and config example: https://docs.pinot.apache.org/operators/operating-pinot/instance-assignment

thankyou 1

8 Views

Open in Slack

Previous Next