Hi Team, Issue: High Off-Heap Memory Usage ("Other" section in NMT) causing OutOfMemoryError and Segments dropping to OFFLINE/ERROR state Detailed Description: We are facing a severe memory issue on our Pinot servers where they climb to maximum memory limits within minutes of a restart. The overhead is entirely off-heap. To investigate, we enabled Native Memory Tracking (NMT) with

detail

mode (

-XX:NativeMemoryTracking=detail

). Looking at the

VM.native_memory.detail

output, the memory is heavily concentrated in the Other section, while the heap looks completely normal: > Other (malloc=41940616KB #435) (~40 GB) For comparison, we checked our Elasticsearch servers under similar loads, and their off-heap/buffer usage remains minimal (only a few MBs out of 30GB available): JSON

{
  "count": 243,
  "used_in_bytes": 23122953,
  "total_capacity_in_bytes": 23122951
}

When checking the Pinot Server logs during these crashes, we see

java.lang.OutOfMemoryError

exceptions being thrown during segment completion protocols: Plaintext

INFO [ServerSegmentCompletionProtocolHandler] [table_name__partition__sequence__timestamp] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"isSplitCommitType":true,"status":"PROCESSED"} for http://<controller-host>:9000/segmentStoppedConsuming?reason=java.lang.OutOfMemoryError&streamPartitionMsgOffset=29329220173&instance=Server_<server-host>_8098&name=table_name__partition__sequence__timestamp

Right after these OOM errors occur, the affected segments transition into an OFFLINE or ERROR state. It feels like

mmap

is either not being effectively utilized, or there is a major native memory leak associated with the "Other" category during data consumption/segment allocation. A few questions for the community: 1. Has anyone seen Apache Pinot allocations balloon specifically in the NMT "Other" category (which usually points to direct

malloc

calls from JNI/native code or specific system allocations)? 2. Could this be related to a misconfiguration in how Pinot utilizes memory-mapped files (

mmap

) versus direct memory allocation for real-time segments? 3. We are using Pinot 1.4. Considering that it uses below setting by default.

# Use off-heap for consuming segments (keep true)
pinot.server.instance.realtime.alloc.offheap=true

# Use MMAP (not direct) for that off-heap allocation (key setting)
pinot.server.instance.realtime.alloc.offheap.direct=false

4. Its not MMAP is not in use at all. It is but major workload is going to direct mem.

PID="`pgrep -f "pinot-server.conf"`" ; grep -E "VmRSS|RssAnon|RssFile" /proc/$PID/status

VmRSS: 81,275,588 kB ≈ 81.28 GB (≈ 77.51 GiB)
RssAnon: 67,669,292 kB ≈ 67.67 GB (≈ 64.53 GiB)
RssFile: 13,605,948 kB ≈ 13.61 GB (≈ 12.98 GiB)

5. What are the best triage steps or configurations to tune to prevent segment consumption from blowing up native memory within minutes? Any guidance or pointers would be greatly appreciated!

Based on the out of memory error message log that you mention, the out of memory situation is not during segment completion. Instead, the thread that tries to consume rows from underlying stream gets an OOM when trying to allocate memory for the rows. It probably happens at different stages for different consuming threads. The server simply reports this fact to the controller and that is the message that you are seeing (the report). Here are some things to think about:

We are evaluating Pinot vs ES using same hardware spec. Ans 1 : Pinot Server Buffer Pool Mapped Memory Used is always at 160GB Mem. any thing specific you are looking for. Ans 2: We havent enabled read path, just scaling for writes. Ans 3: Not all replicas. And its random. • Servers are totally 200 VMs and 160 of them are consuming. • Each server is 200G RAM and 36CPU , 12TB NVME. • load avg is somewhere between 3 to 10. We are evaluating Pinot vs ES using same hardware spec. One more. Do you see if this table definition has any redflag.

{
  "REALTIME": {
    "tableName": "gaming_events_REALTIME",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "replication": "2",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "15",
      "replicasPerPartition": "2",
      "timeColumnName": "eventTime",
      "completionConfig": {
        "completionMode": "DEFAULT"
      },
      "minimizeDataMovement": false
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "GAMING_AWARE",
      "tagOverrideConfig": {}
    },
    "tableIndexConfig": {
      "rangeIndexColumns": [
        "eventTime"
      ],
      "enableDefaultStarTree": false,
      "nullHandlingEnabled": true,
      "sortedColumn": [
        "eventTime"
      ],
      "noDictionaryColumns": [
        "gamePayload",
        "gamePayloadRaw"
      ],
      "invertedIndexColumns": [
        "gameClient",
        "eventData.playerId",
        "eventData.loadoutVariantId",
        "matchGroupKey",
        "eventData.skillModelExperimentDetails",
        "eventData.itemSkuId",
        "eventData.regionHost",
        "eventData.gameMode",
        "eventData.eventPassId",
        "eventData.accountId",
        "eventData.sessionMeta.matchBucketContexts.restrictions.signal",
        "eventData.matchmakingOutcome.rank",
        "eventData.rewardItems",
        "eventData.uiTemplate.templateParams.bannerText.nav.deepLinkAndroid",
        "eventData.telemetrySignal",
        "eventData.eventTuningExperimentDetails",
        "eventData.matchDropReasonMap",
        "eventData.inventoryItemId",
        "eventData.abilityId",
        "eventData.gameBuildId",
        "eventData.platformPlayerId",
        "eventData.matchDropReasonCode",
        "eventData.crafterId",
        "eventData.disconnectReason",
        "eventData.storeCatalogId",
        "eventData.rewardManifest.uiTemplates.variantId",
        "eventData.inventoryItemName",
        "eventData.matchmakingModelId",
        "eventData.matchmakingOutcome.modelId",
        "eventData.itemName",
        "eventData.matchmakingExperimentDetails",
        "eventData.matchmakingOutcome.strategy",
        "eventData.uiTemplate.templateParams.footerNav.deepLinkAndroid",
        "eventData.accountShardId",
        "eventData.matchmakingStrategy",
        "eventName",
        "calendar.dayOfMonth",
        "calendar.minuteOfHour",
        "calendar.hourOfDay",
        "calendar.monthOfYear",
        "calendar.year",
        "calendar.secondOfMinute"
      ],
      "bloomFilterColumns": [
        "eventData.itemSkuId",
        "eventData.playerId",
        "eventData.accountId",
        "eventData.platformPlayerId",
        "eventData.inventoryItemId",
        "eventData.abilityId",
        "eventData.eventPassId",
        "eventData.loadoutVariantId",
        "eventData.storeCatalogId",
        "eventData.matchmakingModelId"
      ],
      "columnMajorSegmentBuilderEnabled": true,
      "aggregateMetrics": false,
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "loadMode": "MMAP",
      "enableDynamicStarTreeCreation": false,
      "skipSegmentPreprocess": false,
      "optimizeDictionary": false,
      "optimizeDictionaryForMetrics": false,
      "optimizeDictionaryType": false,
      "noDictionarySizeRatioThreshold": 0.85
    },
    "metadata": {},
    "quota": {},
    "task": {
      "taskTypeConfigsMap": {
        "MergeRollupTask": {
          "5gb.mergeType": "concat",
          "5gb.bucketTimePeriod": "1h",
          "5gb.bufferTimePeriod": "12h",
          "5gb.maxNumRecordsPerSegment": "12000000",
          "5gb.maxNumRecordsPerTask": "60000000",
          "5gb.maxNumParallelBuckets": "5",
          "5gb.retentionExpiryBufferPeriod": "6h"
        }
      }
    },
    "routing": {
      "segmentPrunerTypes": [
        "time"
      ]
    },
    "instanceAssignmentConfigMap": {
      "CONSUMING": {
        "tagPoolConfig": {
          "tag": "GAMING_AWARE_REALTIME",
          "poolBased": true,
          "numPools": 0
        },
        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 0,
          "numReplicaGroups": 2,
          "numInstancesPerReplicaGroup": 0,
          "numPartitions": 0,
          "numInstancesPerPartition": 0,
          "minimizeDataMovement": false
        },
        "partitionSelector": "FD_AWARE_INSTANCE_PARTITION_SELECTOR",
        "minimizeDataMovement": false
      },
      "COMPLETED": {
        "tagPoolConfig": {
          "tag": "GAMING_AWARE_REALTIME",
          "poolBased": true,
          "numPools": 0
        },
        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 0,
          "numReplicaGroups": 2,
          "numInstancesPerReplicaGroup": 0,
          "numPartitions": 0,
          "numInstancesPerPartition": 0,
          "minimizeDataMovement": false
        },
        "partitionSelector": "FD_AWARE_INSTANCE_PARTITION_SELECTOR",
        "minimizeDataMovement": false
      }
    },
    "ingestionConfig": {
      "streamIngestionConfig": {
        "streamConfigMaps": [
          {
            "streamType": "kafka",
            "stream.kafka.topic.name": "gaming-events",
            "stream.kafka.broker.list": "broker1:9093,broker2:9093,broker3:9093,broker4:9093",
            "stream.kafka.consumer.type": "lowlevel",
            "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
            "realtime.segment.flush.threshold.time": "1h",
            "realtime.segment.flush.threshold.segment.size": "500M",
            "realtime.segment.serverUploadToDeepStore": "true",
            "realtime.segment.flush.threshold.rows": "0",
            "realtime.segment.offHeap": "true",
            "realtime.segment.flush.threshold.segment.size.jitter": "0.10",
            "realtime.segment.direct.realtime.offheap.allocation": "false",
            "sasl.mechanism": "SCRAM-SHA-256",
            "security.protocol": "SASL_SSL",
            "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username='REDACTED' password='REDACTED';",
            "stream.kafka.metadata.populate": "true",
            "<http://stream.kafka.consumer.prop.metadata.max.age.ms|stream.kafka.consumer.prop.metadata.max.age.ms>": "900000"
          }
        ],
        "columnMajorSegmentBuilderEnabled": true,
        "pauselessConsumptionEnabled": false,
        "trackFilteredMessageOffsets": false,
        "enforceConsumptionInOrder": false,
        "useIdealStateToCalculatePreviousSegment": false,
        "disasterRecoveryMode": "DEFAULT"
      },
      "filterConfig": {
        "filterFunction": "strcmp(gameClient, 'sample_game') != 0"
      },
      "transformConfigs": [
        {
          "columnName": "calendar.year",
          "transformFunction": "year(toTimestamp(plus(\"eventTime\", 19800000)))"
        },
        {
          "columnName": "calendar.monthOfYear",
          "transformFunction": "month(toTimestamp(plus(\"eventTime\", 19800000)))"
        },
        {
          "columnName": "calendar.dayOfMonth",
          "transformFunction": "dayOfMonth(toTimestamp(plus(\"eventTime\", 19800000)))"
        },
        {
          "columnName": "calendar.hourOfDay",
          "transformFunction": "hour(toTimestamp(plus(\"eventTime\", 19800000)))"
        },
        {
          "columnName": "calendar.minuteOfHour",
          "transformFunction": "minute(toTimestamp(plus(\"eventTime\", 19800000)))"
        },
        {
          "columnName": "calendar.secondOfMinute",
          "transformFunction": "second(toTimestamp(plus(\"eventTime\", 19800000)))"
        }
      ],
      "complexTypeConfig": {
        "delimiter": ".",
        "collectionNotUnnestedToJson": "ALL"
      },
      "schemaConformingTransformerConfig": {
        "enableIndexableExtras": true,
        "indexableExtrasField": "gamePayload",
        "enableUnindexableExtras": true,
        "unindexableExtrasField": "gamePayloadRaw",
        "unindexableFieldSuffix": "_noindex",
        "fieldPathsToDrop": [],
        "fieldPathsToSkipStorage": [
          "rawMessage"
        ],
        "columnNameToJsonKeyPathMap": {},
        "mergedTextIndexField": "__mergedTextIndex",
        "useAnonymousDotInFieldNames": true,
        "optimizeCaseInsensitiveSearch": false,
        "reverseTextIndexKeyValueOrder": true,
        "mergedTextIndexDocumentMaxLength": 32766,
        "mergedTextIndexBinaryDocumentDetectionMinLength": 512,
        "mergedTextIndexPathToExclude": [],
        "mergedTextIndexPrefixToExclude": [],
        "fieldsToDoubleIngest": [],
        "jsonKeyValueSeparator": ":",
        "mergedTextIndexBeginOfDocAnchor": "",
        "mergedTextIndexEndOfDocAnchor": "",
        "fieldPathsToPreserveInput": [],
        "fieldPathsToPreserveInputWithIndex": []
      },
      "continueOnError": true,
      "retryOnSegmentBuildPrecheckFailure": false,
      "rowTimeValueCheck": false,
      "segmentTimeValueCheck": true
    },
    "isDimTable": false
  }
}