Couple open questions about upsert tables and the metadataTTL config: 1. If no

metadataTTL

is set, do primary keys then expire and free up memroy at the end of the data TTL (set by the

segmentsConfig

retention settings)? Or do primary keys live forever in this case? 2. Should

metadataTTL

always be set to longer than the segment config retention setting? I imagine if you went to update a record that fell out of the

metadataTTL

, but the old segment data was still around, you could have duplicate data at that point?

Hello, I want row based TTL on a particular table. Could anyone suggest any solution. Thanks.

I have a table on production, it was not able to commit the consuming segments from last 3 days, my kafka has retention of 2 days, so now i have deleted my consuming segments and tried to invoking the

RealtimeSegmentValidationManager

task, after that also triggered this api tables/{{table_name}}/resumeConsumption?consumeFrom=largest still consuming segments are not spawning. Please help to resolve this

I noticed that the Pinot server's /debug/memory/offheap API shows 512MB mmap files for each real-time consuming segment (path: data/table_name/consumers/segment_name) The same was mentioned in the javadoc of MmapMemoryManager

We create files of length 0.5g (or the requested buffer length, whichever is higher), and map areas of the file for each allocation request within a segment

Can someone confirm if at least 512MB is allocated per Kafka partition consumer, and is there a way to configure this size to reduce RES memory usage incase of high partition count ?

Hello we are observing wrong result for below query select dateTrunc('hour', 1765295114678, 'MILLISECONDS', 'America/New_York', 'MILLISECONDS') AS original_event_time_hours_buckets2,dateTrunc('hour', 1765295114678, 'MILLISECONDS', 'Europe/Berlin', 'MILLISECONDS') AS original_event_time_hours_buckets from test limit 10 its giving us same Epoch milliseconds as UTC. 1765292400000 1765292400000 it should have different epoch milliseconds for different timezone.How ever ASIA/Kolkota timezone is giving correct result. is it bug for timezone aware query for pinot?

can you please help me?

For multi-stream ingestion, there was a bug in KafkaConsumerFactory (using partitionGroupId instead of streamPartitionGroupId). I'm assuming the fix in PR #15094 isn't in 1.3.0 since it's missing from the release tag and was opened after the Feb 17 release date. Should we explicitly mention in the 1.3.0 release doc that Kafka multi-topic ingestion is broken in this version? Let me know if I'm missing something

Hi Team, we are using

Apache Pinot 1.2.0

and we are facing a scaling issue with realtime upserts enabled for our use case, due to the nature of the primary key the updates can fall in after the expiration window of the

metadataTTL

defined (~8hrs) in our table config, since we cannot keep infinite retention so we are not able to show accurate information on our views due to duplicate entries being recorded We are thinking to write a custom minion job, which should do stuffs like 1. Fetch all such duplicate entries within a time window 2. Only keep the latest one and mark others as invalid so that they can be taken up for deletion in future We wanted to get community help as we didn't find any resources which provides such functionality open source , can anyone help with any references or starting point so that it can help us to use pinot efficiently cc: @Abhigyan Bharati @Xiang Fu

Hi Team, I use Apache Pinot 1.4.0 for explorational analysis of raw events (small time windows) and for upserts based on primary key. The raw events are flat json format (~3.000 Events/second). The problem is that the attribute names are containing "-" like "event-timestamp" or "event-id". I guess Pinot does not like "-" in attribute names. My first idea was to use Apache Nifi for transformation and replace "-" with "_". Nifi seems not to be the right solution for this use case. What do you think would be an efficient solution (3.000-5.000Events/second)? Thank you :)

Hi Team, I have an upsert table with uniq key (STRING, 64 chars). Ingest rate ~4–5k events/sec. Quick math for the raw ID payload: 64 bytes per row 86,400 sec/day 4–5k rows/sec → ~345–432M rows/day • ⇒ ~22–28 GB/day (≈26 GB/day avg) only for raw uniqId bytes My question: how do you roughly estimate RAM usage for upsert/dedup in Pinot (PK → docId map, validDocIds, etc.) given: • key type: 64-char STRING • high cardinality • hot window target: 3 days or more if it passible Any rule-of-thumb or example numbers would help

Hi, I am looking to consider Pinot for our company however, I am concerned I can not see anything about release schedules either in the opensource or closed source version - not any policies on security patches / updates? Have I missed a resource in my search? I notice until last year releases were roughly every 3-6 months but there seems to have been nothing since September 25 - so I am guessing that means a main release is due but what also about security updates and patches inbetween? Thanks

Here are the recordings and slides from meetup Apache Pinot: What’s Advancing Across Query, Ingestion, and the Core Engine. Recording: Here is the recording of the session Slides: https://docs.google.com/presentation/d/1Eiwx8cOXjZfGayX3nLV6ixBJkqLOgwAzGJ0_jMrZp00/edit?usp=sharing

Cross-posting https://apache-pinot.slack.com/archives/C013WKLT5T7/p1773722199562819

Hi Team, does pinot supports multi region (Live-Live within US) deployment ?

Another question, what is the recommendation for migrating realtime segments from one pinot server to another (different cluster) ? For migrating offline segments we are using segment uri push method. Please note that the migration should be idempotent.

Is there any way to update hdfs uri (segment.download.url) for all segments in a pinot table at a time? I would like to change hdfs hardcoded namenode uri to NN HA nameservice. cc: @Mayank @Xiang Fu @Jackie

Hey Pinot team 👋 Slack is recently using Pinot upsert to build the realtime table Could you please you share the recommended metrics to evaluate if the realtime cluster is in the healthy status thanks Based on what we know, heap utilization could be a critical metrics to validate is the node is overwhelming and we can confirm that our heap utilization is under 80% based on this calculation

jvm_memory_bytes_used{instance=~'.*staging.*',area="heap"} / on(instance) group_left jvm_memory_bytes_max{instance=~"pinot-.*",instance=~".*staging-.*",instance=~".*realtime-.*",area="heap"} * 100

cc: @Neeraja Sridharan @Sai Tarun Tadakamalla

Hi team 👋 Pinot 1.4.0 — getting

BrokerResourceMissingError (410)

on a REALTIME table after adding this to broker config:

pinot.broker.adaptive.server.selector.enable.stats.collection=true
pinot.broker.adaptive.server.selector.type=HYBRID

After updating the config, I am encountering the following error while running the query: Error Code: 410 (BrokerResourceMissingError) BrokerResourceMissingError What am I missing here? Observations: • All 3 brokers:

idealState: ONLINE

but

externalView: ERROR

for this table only • Other tables querying fine •

rebuildBrokerResourceFromHelixTags

returns

"ideal state is the same"

@Mayank @Xiang Fu do we have any tool or documentation for migrating Druid to Pinot?

Hello team We have offline table in pinot and that table has a country name and transaction. We want to set a retention period per country.for example india we want 6 days and if japan we want to set 10 days .how can we do it

Hi all, Apache Pinot 1.5.0 has been released! https://github.com/apache/pinot/releases/tag/release-1.5.0 https://pinot.apache.org/download/ https://downloads.apache.org/pinot/apache-pinot-1.5.0/ https://hub.docker.com/layers/apachepinot/pinot/1.5.0

Hi, "I am setting up Apache Pinot 1.4.0 with a REALTIME table consuming from Kafka. My understanding is that as Pinot consumes messages from Kafka, the incoming row data is written to

.0

files (in the consumers directory) which act as a Write-Ahead Log (WAL) buffer for the currently consuming segment. This continues until a flush is triggered, at which point the

.0

file is sealed, converted into an immutable Pinot segment (columnar format with indexes), and a new

.0

file is created for the next consuming segment. With this understanding, I noticed that when the server starts, these

.0

files are created immediately before any data arrives — one per Kafka partition. For example for a table with 3 partitions I see them under:

data/server/index/table_REALTIME/consumers/

-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__0__0__20260421T1050Z.0
-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__1__0__20260421T1050Z.0
-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__2__0__20260421T1050Z.0

When I check with

ls -lh

they always show 512MB regardless of what I set for

realtime.segment.flush.desired.size

or

realtime.segment.flush.threshold.rows

. However when I check with

du -sh

they only show 128KB:

128K    table__0__0__20260421T1050Z.0
128K    table__1__0__20260421T1050Z.0
128K    table__2__0__20260421T1050Z.0

My questions are: 1. Is my understanding correct that

.0

files act as WAL buffers for incoming Kafka data before segment commit? 2. Is 512MB a hardcoded pre-allocation size in Pinot regardless of flush settings or can it be configured? 3. Does

flush.desired.size

only control WHEN the file is flushed/sealed, not the pre-declared file size? 4. As real data arrives from Kafka, will

du

grow until it hits

flush.desired.size

and then trigger a flush — meaning the maximum real disk consumed per

.0

file equals

flush.desired.size

? 5. After flush, a new

.0

file is created — does it again show 512MB in

ls

but 128KB in

du

? Also what exactly is stored in those initial 128KB — is it column metadata, index structures, or something else?" @Mayank, @Xiang Fu

Hi team, When will S3-based tiered storage become available? Is there any active development underway?

Hi Everyone, We are working on reducing the EBS volumes attached to pinot servers in our pinot cluster (deployed in AWS EKS) since they are over-provisioned. Is there any industry wide followed practice for such use cases.

Hi Team, Do we have any limit for segement generation task pick files for offline tables to create segemts. Because I tried for 10,000+ files bucket even task is not created. @Xiang Fu FYI

Hi, I am trying optimize a query which brings data from

30 different segments

of a table. I use

timestamp

for the segment partitioning and also setup

rangeindexcolumn

on the same column

'report_date'

. for example select name, total_sales from product_sales where

report_date > '2024-01-20'

and

report_date < '2024-01-27'.

I see that no index has been used for this because

numEntriesScannedInFilter

is not zero. Lets say, I have

100k rows

per segment for

30days

worth of data then takes about `504ms f`or the query. Anyone knows how to optimize or tune any of the stuff?

Hi Team, we are running Pinot 1.5.0 on Kubernetes with multiple high-throughput realtime upsert tables. The Kafka source topic receives roughly ~3k records/sec. Multiple Pinot realtime tables consume independently from this same stream. We see intermittent UpsertCompactionTask failures across several tables. The failures seem to alternate - sometimes one table fails while the others work, then later antother table fails and the first one works again. A recurring concrete failure looks like this: "java.lang.IllegalStateException: No consensus on validDocs across replicas for segment: table_name__partition__sequence__timestamp. Failing task to avoid replica inconsistency. at org.apache.pinot.plugin.minion.tasks.MinionTaskUtils.getValidDocIdFromServerMatchingCrc(MinionTaskUtils.java:375) at org.apache.pinot.plugin.minion.tasks.upsertcompaction.UpsertCompactionTaskExecutor.convert(UpsertCompactionTaskExecutor.java:87) at org.apache.pinot.plugin.minion.tasks.BaseSingleSegmentConversionExecutor.executeTask(BaseSingleSegmentConversionExecutor.java:124) at org.apache.pinot.plugin.minion.tasks.BaseSingleSegmentConversionExecutor.executeTask(BaseSingleSegmentConversionExecutor.java:63) at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:146) at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:115) at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75)" This seems to indicate that Pinot cannot find a consensus on validDocs across replicas for the same completed segment. Cluster shape: - 2 controllers - 6 brokers - 8 servers - 4 stateless minions - 3 ZooKeeper nodes - S3-compatible deep store - Kafka ingestion Cluster sizing / capacity numbers: Controllers: - replicas: 2 - per pod: 2 CPU request / 4 CPU limit - per pod memory: 8 GiB request / 8 GiB limit - per pod JVM heap: Xms 4G / Xmx 6G - total controller CPU: 4 requested / 8 limited - total controller memory: 16 GiB requested / 16 GiB limited - total controller heap: 8G Xms / 12G Xmx Brokers: - replicas: 6 - per pod: 1 CPU request / 2 CPU limit - per pod memory: 9 GiB request / 12 GiB limit - per pod JVM heap: Xms 6G / Xmx 6G - total broker CPU: 6 requested / 12 limited - total broker memory: 54 GiB requested / 72 GiB limited - total broker heap: 36G Servers: - replicas: 8 - per pod: 6 CPU request / 8 CPU limit - per pod memory: 32 GiB request / 48 GiB limit - per pod JVM heap: Xms 16G / Xmx 28G - persistent storage per server: 600 GiB - total server CPU: 48 requested / 64 limited - total server memory: 256 GiB requested / 384 GiB limited - total server heap: 128G Xms / 224G Xmx - total server local persistent storage: 4.8 TiB Minions: - replicas: 4 - stateless minions - per pod: 2 CPU request / 4 CPU limit - per pod memory: 8 GiB request / 10 GiB limit - per pod JVM heap: Xms 4G / Xmx 6G - per pod ephemeral storage: 30 GiB request / 100 GiB limit - task executor threads per minion: 4 - total minion CPU: 8 requested / 16 limited - total minion memory: 32 GiB requested / 40 GiB limited - total minion heap: 16G Xms / 24G Xmx - total minion ephemeral storage: 120 GiB requested / 400 GiB limited - maximum theoretical minion task slots: 4 minions * 4 threads = 16 parallel tasks ZooKeeper: - replicas: 3 - persistent storage per pod: 20 GiB - total ZooKeeper storage: 60 GiB ealtime table layout: - most tables use 16 logical partitions - replicasPerPartition: 2 - consuming assignment uses replica-group based partition assignment - roughly 16 partitions * 2 replicas = 32 consuming segment assignments per table - across 8 servers this is roughly 4 consuming assignments per server per table - with 4 similar realtime tables this can be roughly 128 consuming assignments total, or around 16 per server, depending on the current segment state Segment settings: - realtime segment flush threshold size: mostly 250-300 MB - realtime segment flush threshold time: 1 day - realtime segment flush threshold rows: 0 Table patterns: The most stable table is a FULL upsert latest-state table: - dropOutOfOrderRecord=true - validDocIdsType=IN_MEMORY - invalidRecordsThresholdPercent=5 - tableMaxNumTasks=1 - schedule every 10 minutes - latest-state semantics The less stable tables are PARTIAL upsert aggregation/history tables: - INCREMENT metrics - dropOutOfOrderRecord=false - validDocIdsType=IN_MEMORY - tableMaxNumTasks=2 - schedules staggered every 5 minutes across tables - aggregation/history-style primary keys Current task scheduling: - latest-state FULL upsert table: every 10 minutes, tableMaxNumTasks=1 - monthly PARTIAL aggregation table: every 20 minutes, tableMaxNumTasks=2 - daily PARTIAL aggregation table: every 20 minutes, tableMaxNumTasks=2 - history-like PARTIAL table: every 20 minutes, tableMaxNumTasks=2 The schedules are staggered, but close together: - monthly table: minute 0/20/40 - daily table: minute 5/25/45 - history-like table: minute 10/30/50 - latest-state table: every 10 minutes So the configured maximum task count across these tables is roughly: - latest-state table: 1 - monthly aggregation table: 2 - daily aggregation table: 2 - history-like table: 2 - total configured table task capacity: up to 7 UpsertCompactionTasks However, because schedules are frequent and tasks may run longer than the schedule interval, we suspect that task backlog or overlap could still create contention against the 16 available Minion task slots. Questions: 1. What are the typical causes of this error during UpsertCompactionTask? "...java.lang.IllegalStateException: No consensus on validDocs across replicas for segment ... Failing task to avoid replica inconsistency." 2. Which metrics/logs best show whether this is caused by validDocs mismatch, CRC mismatch, Minion resource pressure, S3/deep-store issues, or Helix/segment replacement timing? 3. Does our cluster sizing look sufficient for this workload, or are Minions/Servers/deep store likely under-provisioned? Any recommendations on how to debug and stabilize UpsertCompactionTask in this setup would be very helpful. Thanks in advance for your help!

I have a question about using replica groups in Pinot. Replica groups help control query fanout, but is there any recommendation for how many servers a single tenant should have before it makes sense to enable replica groups? In my experience, when the number of servers is small, using replica groups actually makes queries slower because the load concentrates on fewer servers. Switching back to balanced segment assignment makes queries faster. I’m not sure at what server count (per tenant) it’s recommended to start using replica groups, and how I should properly observe or measure query fanout to decide when it becomes a problem. Any guidance or best practices on this would be appreciated.

I have been looking into what would be needed to run pinot tests in parallel - it's mostly collisions in networking ports,and test files. Concurrent tests may use same ports or write same files. It runs the entire suite in ~30m on my machine. Is there any interest making PRs for those changes? It can get pretty involved.

fyi