Apache Pinot #troubleshooting

Vadzim Novikau

03/23/2026, 10:36 PM

Hello! We’re running Apache Pinot 1.3.0 consuming from Kafka topics produced by Flink 1.19 KafkaSink with EXACTLY_ONCE delivery guarantee. Pinot tables use Kafka isolation level

read_committed

. On low-volume topics (can be idle for ~30 minutes), during server restart Pinot’s startup readiness using the

FreshnessBasedConsumptionStatusChecker

gets stuck forever and never turns

GOOD

. Freshness-based check does the fallback to offset-based check, but it fails as well. The log repeats with a stable “+1” offset gap, log example in thread. Manually consuming last hour of user messages shows last user record offset is 18895 (no record at 18896 visible to a normal consumer) This blocks server startup readiness and cause restart loops. It looks like it possibly caused by kafka transactional markers/control records at the tail that aren’t visible for consumer, but still advance the offset. We fixed this issue using

realtimeFreshnessIdleTimeoutMs

, but as I understand it's not recommended to enable it in production environment. Is there any better options to let Pinot skip transactional markers instead of enabling

realtimeFreshnessIdleTimeoutMs

? Using

enableRealtimeOffsetBasedConsumptionStatusChecker

produced the same results

Hassan Ait Brik

03/27/2026, 11:08 AM

Hi team, I am running into an issue on Pinot 1.4.0 with a REALTIME upsert table when trying to use UpsertCompactMergeTask. Context I have a REALTIME upsert table with enableSnapshot=true, and I configured UpsertCompactMergeTask. However, the task does not generate any work for a subset of segments. These segments appear in the UI with: • segment.realtime.status = "IN_PROGRESS" • Helix/server state = OFFLINE • no segment.download.url • no segment.realtime.endOffset • no segment.total.docs Example segment metadata:

Copy code

{
  "segment.realtime.numReplicas": "1",
  "segment.creation.time": "1717253407093",
  "segment.flush.threshold.size": "10000",
  "segment.realtime.startOffset": "4317",
  "segment.realtime.status": "IN_PROGRESS"
}

In the UI, the segment replica is shown as OFFLINE. Example segment name:

Copy code

news__4__1__20240601T1450Z

These segments are also not present in deep store. Problem When I trigger UpsertCompactMergeTask, Pinot skips these segments because their download URL is empty. Controller logs:

Copy code

WARN [UpsertCompactMergeTaskGenerator] Skipping segment news__4__1__20240601T1450Z for task as download url is empty
...
INFO [ServerSegmentMetadataReader] Retrieved validDocIds metadata for 0 segments from 1 server requests.
INFO [UpsertCompactMergeTaskGenerator] Finished generating 0 tasks configs for table: news_REALTIME

So the task cannot proceed because these segments are not considered mergeable. My understanding These look like old zombie consuming segments: • still marked as IN_PROGRESS • server state is OFFLINE • no deep store artifact • no download URL • apparently never committed as completed segments Because of that: • UpsertCompactMergeTask skips them • they cannot be reloaded from deep store • they look like stale metadata / orphaned realtime segments Questions 1. Is this a known issue in Pinot 1.4.0? 2. What is the recommended way to repair or clean up these old IN_PROGRESS + OFFLINE realtime segments when they no longer exist in deep store? 3. Should they be removed from metadata manually, or is there a supported repair workflow? 4. Is there any controller/server operation that should convert or clean up these stale segments automatically? Thanks !

francoisa

03/31/2026, 2:55 PM

Hi, quick question about moving data across environments — looking for best practices. I have a Pinot setup with 36 REALTIME tables (all Kafka-ingested) spanning roughly 2k customers. Some customers are small (100–1k rows per table), some are significantly larger. I’ve built an isolated DEV environment with all my tooling (MCP, Trino, etc.) connected to it, and I deliberately don’t want any of those tools to touch prod directly. What I’m trying to solve: selectively copy a single customer’s data from prod to dev, across all tables. Constraints: - Kafka retention in prod is only 7 days, so replaying from the topic isn’t an option for historical data. - Several tables have large blob columns (noDictionary, multi-MB raw JSON stored in attributes/relationships) that are never used in queries — they’re only there for backfill purposes. I don’t need to copy those. - My main concern is not breaking prod with expensive, large-volume exports — even with pagination, I’m wary of putting too much read pressure on the Pinot cluster. Current approach I’m considering: - Query prod via Trino (which sits in front of Pinot), filtering by meta.customer, selecting only the lightweight indexed columns (skipping the heavy blob columns). - Paginate with LIMIT/OFFSET in manageable batches. - Reconstruct the original Kafka JSON message format by inverting the transformConfigs (since I only have the derived columns, not the raw source JSON). - Push the reconstructed messages into the dev Kafka topic using kcat or a Kafka producer, and let dev Pinot ingest them normally. Questions for the community: 1. Is querying through Trino a reasonable way to extract per-customer data without hammering Pinot directly? Any tips on throttling or batching? 2. Are there better patterns for cross-env data sync in Pinot that I’m not seeing? (Segment-level copy seems too coarse since segments aren’t partitioned by customer.) Thanks in advance!

Raghavendra M

04/02/2026, 9:30 AM

Is there any way to update hdfs uri (segment.download.url) for all segments in a pinot table at a time? I would like to change hdfs hardcoded namenode uri to NN HA nameservice. cc: @Mayank @Xiang Fu @Jackie

Hassan Ait Brik

04/13/2026, 8:10 AM

Hi Pinot team, I’m seeing another UpsertCompactMergeTask failure on Pinot 1.4.0 The task fails during the upload step with this stacktrace:

Copy code

org.apache.pinot.spi.utils.retry.RetriableOperationException:
org.apache.pinot.common.exception.HttpErrorStatusException:
Got error status code: 400 (Bad Request) with reason: ""
while sending request:
/segments?enableParallelPushProtection=true&tableName=success_board&tableType=REALTIME
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:63)
	at org.apache.pinot.plugin.minion.tasks.SegmentConversionUtils.uploadSegment(SegmentConversionUtils.java:140)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.pushSegment(BaseMultipleSegmentsConversionExecutor.java:538)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:316)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:84)

This looks similar to another UpsertCompactMergeTask failure I had before: • task generation succeeds • source segments seem valid • failure happens when uploading the merged segment back to the controller • the request includes enableParallelPushProtection=true Questions: 1. What are the common reasons for the controller to return 400 Bad Request on /segments?enableParallelPushProtection=true... for UpsertCompactMergeTask ? 2. Does this usually indicate: • parallel push conflict, • segment lineage changed while the task was running, • or another controller-side validation failure ? 3. Is there a recommended way to debug the exact controller-side reason when the response body is empty ? 4. Are there recommended limits for maxNumSegmentsPerTask / maxNumRecordsPerTask to reduce the chance of this happening? Thanks!

Raghavendra M

04/14/2026, 6:51 AM

@Mayank @Xiang Fu do we have any tool or documentation for migrating Druid to Pinot?

Badhusha Muhammed

04/15/2026, 9:14 AM

In Pinot 1.4.0 , It does support multi topic ingestion from same kafka cluster right ...? When created table config , with

streamConfigMaps

Pinot table is only consuming from the first config in the list

Copy code

"streamConfigMaps" : [
    { "stream.kafka.topic.name": "topic_1", ... },
    { "stream.kafka.topic.name": "topic_2", ... }
]

This result in consuming only from

topic_1

When looking at log , i can see that, the

topic_partition

that is trying to consume is going with

Segment Partition ID

(In this case the second topic segment is created with largest partition ID - 10000, https://github.com/apache/pinot/issues/17260) Log for correctly consuming topic

Copy code

2026/04/15 02:19:58.163 INFO [KafkaConsumer] [multi_topic_test_1__0__0__20260414T2017Z] [Consumer clientId=multi_topic_test_1_REALTIME-mulit_topic_test_2-0, groupId=null] Seeking to offset 6 for partition mulit_topic_test_2-0

Seeking to offset 6 for partition mulit_topic_test_2-0

Correct Log for not consuming topic

Copy code

2026/04/15 02:19:56.000 INFO [KafkaConsumer] [multi_topic_test_1__10000__0__20260414T2017Z] [Consumer clientId=multi_topic_test_1_REALTIME-mulit_topic_test_1-0, groupId=null] Seeking to offset 0 for partition mulit_topic_test_1-10000

Seeking to offset 0 for partition mulit_topic_test_1-10000

???????? Is this is a bug ..? Partition part of

topic_partition

is picking from

Segment Partition ID

...?

Badhusha Muhammed

04/15/2026, 9:14 AM

cc: @Mayank, @Xiang Fu Can you please help here .

Richa Kumari

04/15/2026, 1:55 PM

Hi Pinot Team , We are migrating from Elasticsearch to Pinot . We use cursor pagination (POST /query/sql?getCursor=true) on V1 to fetch large result sets including the requestId column (primary key). We originally used V2 (/query) for cursor, but V2's multi-stage engine fails when requestId is in the SELECT clause — the query is cancelled during plan setup (EMPTY_MAILBOX_SEND, error 503, "CANCELLED: client cancelled") because requestId conflicts with V2's internal gRPC stage-tracking identifier. Alias, CAST, and CONCAT wrapping all failed. Per official docs, cursor is a V1 feature, so we switched to /query/sql?getCursor=true. The initial cursor query works correctly and returns the first page. However, subsequent pagination GETs to /responseStore/{requestId}/results fail intermittently with "Query results for {requestId} not found". Out of 12 concurrent Lambda invocations (1 hour split into 5-min windows), some succeed fully, some fail immediately, and some partially succeed (e.g., 600K of 1.4M records fetched before failure). @Mayank @Xiang Fu

Richa Kumari

04/16/2026, 9:55 AM

@Jinesh Parakh @Mayank @Xiang Fu Hi Pinot team, We're running into a blocking issue with the V2 multi-stage engine and need guidance. Our Setup: • Pinot cluster with 18 brokers behind a Network Load Balancer (NLB) • Table: urdmCore with requestId as the primary key column • Query volume: ~1.4M+ records per query, using cursor-based pagination • Client: AWS Lambda (Java) The Problem: When we include requestId in the SELECT clause on the V2 multi-stage engine (/query), the query gets cancelled immediately with EMPTY_MAILBOX_SEND / error 503 CANCELLED errors. Omitting requestId from SELECT works perfectly — cursor, pagination, everything functions as expected. We believe this is a namespace collision between V2's internal gRPC stage-tracking (which uses requestId for inter-stage communication) and our column named requestId. requestId is our primary key — we cannot omit it from SELECT. We need it for record-level validation between data sources. We are thinking of creating new column with changed name since pinot columns are immutable , was considering using transform function to get old data of requestId column in new column. Just needed an opinion / heads-up on this approach.

Soon

04/17/2026, 6:00 PM

Hi Pinot team! I'm currently encountering a weird behavior in text index where regex is not matching rows as expected. For example, my query contains

WHERE TEXT_MATCH(indexedString3, '/.*nav#primary-menu.*/')

and its not matching a row with

nav#primary-menu

. My theory this is due to that special character

and that standard analyzer breaks them up into tokens like

nav

and

primary-menu

and never keep them as a whole. Would be greatly helpful if someone could confirm my theory! Thanks in advance!

Shubham

04/29/2026, 6:18 AM

Hi Team. We are seeing high latency for the following query : SELECT cevalDec, vevalDec, DISTINCTCOUNT(subjectId) AS Count FROM ModelScoreVariableTable WHERE ekey = 'ee801a97-aae1-490d-ac7f-d5556060f758' GROUP BY cevalDec, vevalDec ORDER BY cevalDec, vevalDec LIMIT 1000 Can you let us know if we can optimize it? Attaching other information in thread.

piby

04/30/2026, 9:16 AM

Hi Pinot Community! 👋 We are refining our query standards to ensure we avoid implicit casting and leverage indexes properly. Consider this example of column my_timestamp_col of type TIMESTAMP

Copy code

WHERE my_timestamp_col > '2025-11-02T00:00:00+01:00‘

This syntax works but it takes a very long time. I am assuming either Pinot is implicitly casting my_timestamp_col to string and doing string comparison or indexes are getting ignored due to type mismatch. When we convert our datetime string to TIMESTAMP type exactly, our query performance improves significantly. However, I’m curious which of these three approaches is recommended for filtering TIMESTAMP columns, especially regarding the Multi-Stage Engine: Option A (ANSI SQL CAST):

Copy code

WHERE my_timestamp_col > CAST('2025-11-02T00:00:00+01:00' AS TIMESTAMP)

Option B (Explicit Functions):

Copy code

WHERE my_timestamp_col > toTimestamp(fromIso8601('2025-11-02T00:00:00+01:00'))

Option C (Standard Literal - This does not support timezones - UTC only ):

Copy code

WHERE my_timestamp_col > TIMESTAMP '2025-11-02 00:00'

Our specific questions: Performance: Is any one of these more "index-friendly" than the others? Timezones: Does CAST (Option A) reliably handle the +01:00 offset, or should we stick to fromIso8601 (Option B) for anything non-UTC? Engine Preference: With the move toward the Multi-Stage Engine, is there a preferred syntax for long-term compatibility? (we only use multi stage engine) Would love to hear your thoughts on the "cleanest" way to handle this in production. Thanks!

piby

04/30/2026, 3:06 PM

Hey, We have issues with realtime/hybrid tables whenever we perform rolling restarts on servers. We are using Pinot 1.4.0. How can we make it more reliable? We do have 3 replicas per segment. So if one server is down, we do expect it to take the data from replicas. Furthermore, no query works on the table if only one segment is down. If we just want to query the latest data and some old segment is down, we do get the error.

Copy code

When multistage is enabled

query > SELECT * FROM my_table LIMIT 10;

useMultiStage=true
usePhysicalOptimizer=true

DatabaseError: {'errorCode': 150,
 'message': 'SQLParsingError: Error composing query plan: No enum constant '
            'org.apache.pinot.spi.config.table.TableType.my_table. No '
            'enum constant '
            'org.apache.pinot.spi.config.table.TableType.my_table'}

useMultistageEngine=false
usePhysicalOptimizer=false

DatabaseError: {'errorCode': 305,
 'message': '3 segments unavailable '
            '[my_table__2__3373__20260430T0524Z, '
            'my_table__2__3380__20260430T1225Z, '
            'my_table__2__3180__20260422T0357Z, '], with routing policy: '
            'balanced [realtime], balanced [offline]'}

Rajat

05/25/2026, 11:03 AM

Hey Pinot folks 👋 I'm trying to validate a PARTIAL upsert use case for our shipmentMerged table and ran into behavior I can't reconcile with the docs. Would love a quick sanity check on what's actually intended. Setup: mode: PARTIAL, nullHandlingEnabled: true, comparison column s_updated_at. Use case: dual-writer on the same table. One Flink pipeline emits full shipment records (all ~80 fields populated). A second pipeline emits partial records carrying only 2 NDR fields (is_ndr, no_of_attempts) — the other 78 columns are Avro-null. Goal: partial records update only the 2 NDR columns; full records update everything; intentional NULLs from source MySQL propagate to Pinot. What I observed on the test cluster:

Copy code

Strategy   | Non-null incoming  | Null incoming (nullHandlingEnabled:        true)
-----------+--------------------+-----------------------------------------
IGNORE     | Keeps existing     | Keeps existing
           | (never updates     |
           |  after first       |
           |  insert)    
-----------+--------------------+-----------------------------------------       
OVERWRITE  | Takes new          | Framework preserves existing
           |                    | (null skipped before merger)

If this is correct, then: 1. There's no strategy combination that lets a single PARTIAL table support our dual-writer pattern AND propagate intentional NULLs from source. The two requirements collide on how "incoming null" is interpreted. 2. IGNORE is unusable for any column that ever needs to update — feels less like "ignore" and more like "freeze after first insert." 3. OVERWRITE + nullHandlingEnabled: true is effectively "OVERWRITE_NON_NULL" — useful, but it means partial-upsert can't intentionally clear a column to null. Questions: 1. Is the table above the documented intended behavior, or am I missing a config knob? 2. Is there a known pattern in Pinot for "update on non-null AND clear to null when source genuinely cleared"? (Other than nullHandlingEnabled: false, which loses Avro null semantics globally.) 3. Any plan for a null-aware variant strategy in upcoming releases? We've reverted to FULL upsert for our prod path. Just want to make sure I'm not under-using PARTIAL. Thanks 🙏 C.C: @Mayank @Xiang Fu @Jackie

Hassan Ait Brik

05/27/2026, 12:37 PM

Hi Pinot community, I’m trying to clarify the expected behavior of

UpsertCompactMergeTask

with

validDocIdsType=SNAPSHOT

IN_MEMORY

. We have a REALTIME upsert table on Pinot 1.4.0 with:

Copy code

"upsertConfig": {
  "mode": "FULL",
  "snapshot": "ENABLE",
  "enableSnapshot": true
}

At the beginning of May, we updated this table config to add

UpsertCompactMergeTask

. Since then, the task has worked partially: we reduced the table from around 20K segments to around 3K segments. However, a lot of old completed segments still cannot be merged. From what we see, the task is not able to retrieve their

validDocIds

using

SNAPSHOT

. When triggering the task manually, the controller logs show something like:

Copy code

Retrieved validDocIds metadata for 15 segments from 96 server requests
Finished generating 0 tasks configs

But we still have many old completed segments from 2023 to 2025 that look like they should be compactable. I tested the

validDocIdsMetadata

endpoint on one old segment:

Copy code

validDocIdsType=SNAPSHOT -> []
validDocIdsType=IN_MEMORY -> returns metadata correctly
validDocIdsType=IN_MEMORY_WITH_DELETE -> []

Example with `IN_MEMORY`:

Copy code

{
  "segmentName": "daily_activity__2__0__20230427T1137Z",
  "totalValidDocs": 4,
  "totalInvalidDocs": 0,
  "totalDocs": 4,
  "segmentCrc": "2205091866",
  "validDocIdsType": "IN_MEMORY",
  "serverStatus": "GOOD"
}

So it looks like the upsert state exists in memory on the realtime servers, but many historical segments do not have

validDocIds

snapshots. From the docs, I understand that: •

SNAPSHOT

is the default

validDocIdsType

for

UpsertCompactMergeTask

•

SNAPSHOT

requires

enableSnapshot=true

• snapshots are created/refreshed during the normal upsert / segment commit lifecycle • there does not seem to be a documented command/API to backfill

validDocIds

snapshots for all already-committed historical segments My questions are: 1. Is this expected behavior for old completed segments that existed before we added the compaction task / before snapshots were generated for them? 2. Is there any supported way to force Pinot to generate

validDocIds

snapshots for all existing completed segments? 3. If not, is the recommended approach to run UpsertCompactMergeTask temporarily with:"validDocIdsType": "IN_MEMORY" to compact the remaining historical backlog, and then switch back to

SNAPSHOT

for steady state? 4. Are there any risks or caveats with using

IN_MEMORY

for this remaining backlog, assuming the servers are stable and the table is not being restarted/rebalanced? Thanks!

Badhusha Muhammed

05/27/2026, 6:06 PM

I know its after a long time, But just wanted to update one thing . > On this specific Pinot server, all tables are upsert-enabled with a combined disk utilization of 612GB. What could be causing the heap to spike to 700GB? During Pinot server startup, the heap fills up rapidly, leading to continuous Full GC pauses and OOM failures. Upon debugging, we got to know that the

validDocIds

snapshot was missing from many segments. Assumption 1. Dataset: We are running a 10 TB Upsert table containing millions of rows with a high-cardinality primary key. 2. Restart Behavior: When a server hosting an Upsert table restarts, it must rebuild the in-memory Upsert Metadata Manager (the hash map tracking

PrimaryKey -> RecordLocation

). 3. Segments with Snapshots: The server can efficiently skip entire segments if their maximum timestamp falls outside the metadata TTL, keeping heap usage low. 4. Segments without Snapshots: The server is forced to load the entire segment into the heap to determine which records are valid. A background TTL filter job (running , lets say every 10 minutes) is responsible for cleaning up expired keys. Now i feel like the 4th point is causing this issue of high heap , If yes ( 4th point is correct ) Is there any way we can create snapshot for segments thats are missing them..? If no ( 4th point is wrong ) Can you please let me know, what other reason can be there ..? @Xiang Fu, @Mayank

Senthil Maruthaiappan

05/28/2026, 7:54 PM

Hi, I am trying optimize a query which brings data from

30 different segments

of a table. I use

timestamp

for the segment partitioning and also setup

rangeindexcolumn

on the same column

'report_date'

. for example select name, total_sales from product_sales where

report_date > '2024-01-20'

and

report_date < '2024-01-27'.

I see that no index has been used for this because

numEntriesScannedInFilter

is not zero. Lets say, I have

100k rows

per segment for

30days

worth of data then takes about `504ms f`or the query. Anyone knows how to optimize or tune any of the stuff?

dstawiarz

05/29/2026, 12:15 PM

Hi Team, I am new to Pinot (1.2.0 version) and facing following challenge: currently we have one realtime table on our Pinot cluster. We would like to remove it and recreate with the same name on a daily basis. We often see that although table was successfully deleted, external view is present long after table removal. Only once server is restarted EV seems to be removed. My question would be: • Is there any direct way to remove EV or force Pinot to refresh EV status, • Is it safe to create the table even EV is still visible after deletion or we must wait until EV is no longer present? Thanks!

piby

06/02/2026, 8:02 AM

Hi Team, We had an incident where heap memory of one of our pinot servers was above 90%. This caused downtime in production. How can we prevent it in the future? We are using pinot 1.4.0 currently on k8s and value.yaml is in the comments.

Claudia Asanza

06/03/2026, 9:58 PM

Hi! I tested MergeRollupTask on an OFFLINE APPEND table using: 1month.mergeType = concat 1month.bucketTimePeriod = 1M Expected: daily segments -> one monthly segment Observed: one daily segment with hourly bucket_start_ts values -> 24 hourly merged segments The task completes successfully, but increases the number of segments. Is this expected behavior for a DATE_TIME column with granularity 1:HOURS?

Raghavendra M

06/15/2026, 12:45 PM

Hi @Mayank @Xiang Fu while loading json data to OFFLINE table , am getting below error and segment state showing as ERROR.

Copy code

java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350)
        at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278)
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.OutOfMemoryError: Cannot reserve 132127 bytes of direct buffer memory (allocated: 64424466414, limit: 64424509440)

am actually using spark3 to push data to pinot OFFLINE table. any help here?

Badhusha Muhammed

06/17/2026, 3:32 AM

Hi Team We have an Upset Table in Pinot 1.2.0/1.3.0 version, which has TIMESTAMP dType as timeColumn , But after Pinot upgrade to 1.4.0 , timeColumn does not support TIMESTAMP datatype if TTL is enabled , Which is restricting us to do schema evolutions . Is there any known workaround for this , other than re-creating the table..? @Mayank, @Xiang Fu

samirun shaik

06/25/2026, 8:07 AM

@Mayank @Xiang Fu we recently introduced deepstore in our cluster but we see only newly compelted segments are only reaching deepstore . we did repair and reload segments , and force commit but doesnt seem to get the old segments into deepstore. are we missing anything? i dont see documentations around these details aswell apart form what we have already tried

piby

06/26/2026, 10:56 AM

Hey all, Since migrating 1.4 → 1.5 and pinot-kafka-2.0 → pinot-kafka-3.0, realtime tables consuming from transactional Kafka topics (AWS MSK) get stuck. With 3 partitions × 3 replicas, one consuming segment per table wedges in an internal re-seek loop and never advances. - Reset segment: does not help - Pause / resume: does not help - Adding

stream.kafka.isolation.level: read_committed

(and confirming the table picks it up): does not help — segment still gets stuck - Only workaround: delete table, set ``auto.offset.reset=smallest``, backfill. Works until the next segment rolls — then a new segment gets stuck again at a commit boundary. It only affects transactional topics. Non-transactional topics are fine. Do you have any suggestions for me?

Mannoj

07/10/2026, 7:57 AM

Hi Team, Issue: High Off-Heap Memory Usage ("Other" section in NMT) causing OutOfMemoryError and Segments dropping to OFFLINE/ERROR state Detailed Description: We are facing a severe memory issue on our Pinot servers where they climb to maximum memory limits within minutes of a restart. The overhead is entirely off-heap. To investigate, we enabled Native Memory Tracking (NMT) with

detail

mode (

-XX:NativeMemoryTracking=detail

). Looking at the

VM.native_memory.detail

output, the memory is heavily concentrated in the Other section, while the heap looks completely normal: > Other (malloc=41940616KB #435) (~40 GB) For comparison, we checked our Elasticsearch servers under similar loads, and their off-heap/buffer usage remains minimal (only a few MBs out of 30GB available): JSON

Copy code

{
  "count": 243,
  "used_in_bytes": 23122953,
  "total_capacity_in_bytes": 23122951
}

When checking the Pinot Server logs during these crashes, we see

java.lang.OutOfMemoryError

exceptions being thrown during segment completion protocols: Plaintext

Copy code

INFO [ServerSegmentCompletionProtocolHandler] [table_name__partition__sequence__timestamp] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"isSplitCommitType":true,"status":"PROCESSED"} for http://<controller-host>:9000/segmentStoppedConsuming?reason=java.lang.OutOfMemoryError&streamPartitionMsgOffset=29329220173&instance=Server_<server-host>_8098&name=table_name__partition__sequence__timestamp

Right after these OOM errors occur, the affected segments transition into an OFFLINE or ERROR state. It feels like

mmap

is either not being effectively utilized, or there is a major native memory leak associated with the "Other" category during data consumption/segment allocation. A few questions for the community: 1. Has anyone seen Apache Pinot allocations balloon specifically in the NMT "Other" category (which usually points to direct

malloc

calls from JNI/native code or specific system allocations)? 2. Could this be related to a misconfiguration in how Pinot utilizes memory-mapped files (

mmap

) versus direct memory allocation for real-time segments? 3. We are using Pinot 1.4. Considering that it uses below setting by default.

Copy code

# Use off-heap for consuming segments (keep true)
pinot.server.instance.realtime.alloc.offheap=true

# Use MMAP (not direct) for that off-heap allocation (key setting)
pinot.server.instance.realtime.alloc.offheap.direct=false

4. Its not MMAP is not in use at all. It is but major workload is going to direct mem.

Copy code

PID="`pgrep -f "pinot-server.conf"`" ; grep -E "VmRSS|RssAnon|RssFile" /proc/$PID/status

VmRSS: 81,275,588 kB ≈ 81.28 GB (≈ 77.51 GiB)
RssAnon: 67,669,292 kB ≈ 67.67 GB (≈ 64.53 GiB)
RssFile: 13,605,948 kB ≈ 13.61 GB (≈ 12.98 GiB)

5. What are the best triage steps or configurations to tune to prevent segment consumption from blowing up native memory within minutes? Any guidance or pointers would be greatly appreciated!

Priyanshu Soni

07/22/2026, 2:03 PM

hi does pinot have existing recovery runbooks to recover data from hdfs in case there is data loss from pinot server disks? working out how to recover data in case of cluster wide data loss issues

prasanna

07/28/2026, 9:27 AM

Hi Team, just wanted to ask a very basic question in Pinot 1.5 version do we have a way to manage table retention on a dimension/categorical field other than time based retention that we use today in our pinot 1.3 deployments. example - we have event data coming into our tables via kafka and we have Hybrid tables. with 7 Days RT and 30 Days OF line table. we have a lifecycle for each event at end of which status is Clear State we would like to drop all events that are in cleared state as this table we want to maintain only active events.

Rishabh Sharma

07/30/2026, 5:57 AM

Hi all — a couple of security questions :

1. What tools are people using to scan the Pinot libraries/dependencies for vulnerabilities?

2. Are there any known open/unresolved critical vulnerabilities in Pinot 1.5.1 specifically? If so, are there recommended mitigations or a target release for fixes?

Trying to assess 1.5.1 for a production deployment. Any pointers to a security advisory page or process for reporting/tracking CVEs would also be appreciated. Thanks!

@Xiang Fu, @Mayank

Priyanshu Soni

07/31/2026, 9:34 AM

As a rule of thumb is it discouraged to perform any egress from Pinot servers apart from deep store commits? since it may interfere with ongoing ingestion