Hey, We have issues with realtime/hybrid tables whenever we perform rolling restarts on servers. We are using Pinot 1.4.0. How can we make it more reliable? We do have 3 replicas per segment. So if one server is down, we do expect it to take the data from replicas. Furthermore, no query works on the table if only one segment is down. If we just want to query the latest data and some old segment is down, we do get the error.

When multistage is enabled

query > SELECT * FROM my_table LIMIT 10;

useMultiStage=true
usePhysicalOptimizer=true

DatabaseError: {'errorCode': 150,
 'message': 'SQLParsingError: Error composing query plan: No enum constant '
            'org.apache.pinot.spi.config.table.TableType.my_table. No '
            'enum constant '
            'org.apache.pinot.spi.config.table.TableType.my_table'}

useMultistageEngine=false
usePhysicalOptimizer=false

DatabaseError: {'errorCode': 305,
 'message': '3 segments unavailable '
            '[my_table__2__3373__20260430T0524Z, '
            'my_table__2__3380__20260430T1225Z, '
            'my_table__2__3180__20260422T0357Z, '], with routing policy: '
            'balanced [realtime], balanced [offline]'}

how are you performing “rolling” restarts? Are you doing 1 server at a time or many? the error says you’re using “balanced” routing. So my guess is you are either • taking down too many servers at once and not checking what segments are on what servers before doing so • or you are not correctly configuring “replica groups”, and the data is not actually laid out correctly

We are using kubectl rollout restart command which restart servers one at a time One of our realtime table config is like this

{
  "REALTIME": {
    "tableName": "my_table_REALTIME",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "replication": "3",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "15",
      "timeType": "MILLISECONDS",
      "replicasPerPartition": "1",
      "timeColumnName": "mytimecolumn",
      "segmentPushType": "APPEND",
      "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
      "minimizeDataMovement": false
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "aggregateMetrics": false,
      "enableDefaultStarTree": false,
      "nullHandlingEnabled": true,
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "loadMode": "MMAP",
      "enableDynamicStarTreeCreation": false,
      "columnMajorSegmentBuilderEnabled": true,
      "skipSegmentPreprocess": false,
      "optimizeDictionary": false,
      "optimizeDictionaryForMetrics": false,
      "optimizeDictionaryType": false,
      "noDictionarySizeRatioThreshold": 0.85
    },
    "metadata": {
      "customConfigs": {}
    },
    "ingestionConfig": {
      "streamIngestionConfig": {
        "streamConfigMaps": [
          {
            "key.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
            "realtime.segment.flush.threshold.rows": "0",
            "realtime.segment.flush.threshold.segment.size": "50M",
            "realtime.segment.flush.threshold.time": "1h",
            "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"xxx\" password=\"xxx\";",
            "sasl.mechanism": "SCRAM-SHA-512",
            "security.protocol": "SASL_SSL",
            "stream.kafka.broker.list": "<http://b-2.xxx.amazonaws.com:9096,b-1.xxx.amazonaws.com:9096,b-3.xxx.amazonaws.com:9096|b-2.xxx.amazonaws.com:9096,b-1.xxx.amazonaws.com:9096,b-3.xxx.amazonaws.com:9096>",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
            "stream.kafka.consumer.prop.auto.offset.reset": "largest",
            "stream.kafka.consumer.type": "LOWLEVEL",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
            "stream.kafka.decoder.prop.format": "JSON",
            "stream.kafka.topic.name": "xxx",
            "streamType": "kafka",
            "value.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer"
          }
        ],
        "pauselessConsumptionEnabled": false,
        "columnMajorSegmentBuilderEnabled": true,
        "trackFilteredMessageOffsets": false,
        "enforceConsumptionInOrder": false,
        "useIdealStateToCalculatePreviousSegment": false,
        "disasterRecoveryMode": "DEFAULT"
      },
      "transformConfigs": [],
      "continueOnError": true,
      "retryOnSegmentBuildPrecheckFailure": false,
      "rowTimeValueCheck": true,
      "segmentTimeValueCheck": false
    },
    "isDimTable": false
  }
}

if it’s 1 at a time, that’s tougher to explain. in the lifecycle of server down -> server up, how long does that take? and when are the errors happening and for how long?

We currently have 6 servers each with 100 gb memory , 14 cores and 1 TB storage The cluster is currently managing 100k segments. Each server takes roughly 10-15 mins to come back up so the whole cluster restarted in approximately 1 hr. We do see a lot of errors like these especially for realtime tables.

if server stops at T0 and starts at T15, when are the errors happening in that timeline

I have to check this but I would say in the middle. Right when the server pod transitions from ContainerCreating to Running

we’ve seen issues in the past where a server is healthy, but the broker has not had time to catch up and realize yet. We had to add a 30-60s sleep between server restarts to give the broker more time to recognize servers are back. but it’s still weird to happen with 1 restart at a time especially if you have 3 replicas. it theoretically shouldn’t be possible since 2 servers can at most ever be down even with some overalp

I will run some tests and try to restart the servers on our test cluster and replicate the issue. Will get back to you here

In your ideal state, what are the properties of the offending segments? Check if all the replicas are assigned to different servers.

Update on this issue: So we found out we had only one replica for realtime segments. When the server went down, this segment was not available. We do have

replication = 3

for this realtime table but it still had only one replica. I tried setting

replicasPerPartition = 3

and rebalanced the table and Pinot has now created three replicas. Isn‘t

replicasPerPartition

deprecated? Docs says

Deprecation, use replication

. https://docs.pinot.apache.org/reference/configuration-reference/table#segmentsconfig It is really weird because there are other realtime tables where only replication is defined and it still has 3 replicas