Cross-posting https://apache-pinot.slack.com/archives/C013WKLT5T7/p1773722199562819

Hi Team, does pinot supports multi region (Live-Live within US) deployment ?

Another question, what is the recommendation for migrating realtime segments from one pinot server to another (different cluster) ? For migrating offline segments we are using segment uri push method. Please note that the migration should be idempotent.

Is there any way to update hdfs uri (segment.download.url) for all segments in a pinot table at a time? I would like to change hdfs hardcoded namenode uri to NN HA nameservice. cc: @Mayank @Xiang Fu @Jackie

Hey Pinot team 👋 Slack is recently using Pinot upsert to build the realtime table Could you please you share the recommended metrics to evaluate if the realtime cluster is in the healthy status thanks Based on what we know, heap utilization could be a critical metrics to validate is the node is overwhelming and we can confirm that our heap utilization is under 80% based on this calculation

jvm_memory_bytes_used{instance=~'.*staging.*',area="heap"} / on(instance) group_left jvm_memory_bytes_max{instance=~"pinot-.*",instance=~".*staging-.*",instance=~".*realtime-.*",area="heap"} * 100

cc: @Neeraja Sridharan @Sai Tarun Tadakamalla

Hi team 👋 Pinot 1.4.0 — getting

BrokerResourceMissingError (410)

on a REALTIME table after adding this to broker config:

pinot.broker.adaptive.server.selector.enable.stats.collection=true
pinot.broker.adaptive.server.selector.type=HYBRID

After updating the config, I am encountering the following error while running the query: Error Code: 410 (BrokerResourceMissingError) BrokerResourceMissingError What am I missing here? Observations: • All 3 brokers:

idealState: ONLINE

but

externalView: ERROR

for this table only • Other tables querying fine •

rebuildBrokerResourceFromHelixTags

returns

"ideal state is the same"

@Mayank @Xiang Fu do we have any tool or documentation for migrating Druid to Pinot?

Hello team We have offline table in pinot and that table has a country name and transaction. We want to set a retention period per country.for example india we want 6 days and if japan we want to set 10 days .how can we do it

Hi all, Apache Pinot 1.5.0 has been released! https://github.com/apache/pinot/releases/tag/release-1.5.0 https://pinot.apache.org/download/ https://downloads.apache.org/pinot/apache-pinot-1.5.0/ https://hub.docker.com/layers/apachepinot/pinot/1.5.0

Hi, "I am setting up Apache Pinot 1.4.0 with a REALTIME table consuming from Kafka. My understanding is that as Pinot consumes messages from Kafka, the incoming row data is written to

.0

files (in the consumers directory) which act as a Write-Ahead Log (WAL) buffer for the currently consuming segment. This continues until a flush is triggered, at which point the

.0

file is sealed, converted into an immutable Pinot segment (columnar format with indexes), and a new

.0

file is created for the next consuming segment. With this understanding, I noticed that when the server starts, these

.0

files are created immediately before any data arrives — one per Kafka partition. For example for a table with 3 partitions I see them under:

data/server/index/table_REALTIME/consumers/

-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__0__0__20260421T1050Z.0
-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__1__0__20260421T1050Z.0
-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__2__0__20260421T1050Z.0

When I check with

ls -lh

they always show 512MB regardless of what I set for

realtime.segment.flush.desired.size

or

realtime.segment.flush.threshold.rows

. However when I check with

du -sh

they only show 128KB:

128K    table__0__0__20260421T1050Z.0
128K    table__1__0__20260421T1050Z.0
128K    table__2__0__20260421T1050Z.0

My questions are: 1. Is my understanding correct that

.0

files act as WAL buffers for incoming Kafka data before segment commit? 2. Is 512MB a hardcoded pre-allocation size in Pinot regardless of flush settings or can it be configured? 3. Does

flush.desired.size

only control WHEN the file is flushed/sealed, not the pre-declared file size? 4. As real data arrives from Kafka, will

du

grow until it hits

flush.desired.size

and then trigger a flush — meaning the maximum real disk consumed per

.0

file equals

flush.desired.size

? 5. After flush, a new

.0

file is created — does it again show 512MB in

ls

but 128KB in

du

? Also what exactly is stored in those initial 128KB — is it column metadata, index structures, or something else?" @Mayank, @Xiang Fu

Hi team, When will S3-based tiered storage become available? Is there any active development underway?

Hi Everyone, We are working on reducing the EBS volumes attached to pinot servers in our pinot cluster (deployed in AWS EKS) since they are over-provisioned. Is there any industry wide followed practice for such use cases.

Hi Team, Do we have any limit for segement generation task pick files for offline tables to create segemts. Because I tried for 10,000+ files bucket even task is not created. @Xiang Fu FYI

Hi, I am trying optimize a query which brings data from

30 different segments

of a table. I use

timestamp

for the segment partitioning and also setup

rangeindexcolumn

on the same column

'report_date'

. for example select name, total_sales from product_sales where

report_date > '2024-01-20'

and

report_date < '2024-01-27'.

I see that no index has been used for this because

numEntriesScannedInFilter

is not zero. Lets say, I have

100k rows

per segment for

30days

worth of data then takes about `504ms f`or the query. Anyone knows how to optimize or tune any of the stuff?

Hi Team, we are running Pinot 1.5.0 on Kubernetes with multiple high-throughput realtime upsert tables. The Kafka source topic receives roughly ~3k records/sec. Multiple Pinot realtime tables consume independently from this same stream. We see intermittent UpsertCompactionTask failures across several tables. The failures seem to alternate - sometimes one table fails while the others work, then later antother table fails and the first one works again. A recurring concrete failure looks like this: "java.lang.IllegalStateException: No consensus on validDocs across replicas for segment: table_name__partition__sequence__timestamp. Failing task to avoid replica inconsistency. at org.apache.pinot.plugin.minion.tasks.MinionTaskUtils.getValidDocIdFromServerMatchingCrc(MinionTaskUtils.java:375) at org.apache.pinot.plugin.minion.tasks.upsertcompaction.UpsertCompactionTaskExecutor.convert(UpsertCompactionTaskExecutor.java:87) at org.apache.pinot.plugin.minion.tasks.BaseSingleSegmentConversionExecutor.executeTask(BaseSingleSegmentConversionExecutor.java:124) at org.apache.pinot.plugin.minion.tasks.BaseSingleSegmentConversionExecutor.executeTask(BaseSingleSegmentConversionExecutor.java:63) at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:146) at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:115) at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75)" This seems to indicate that Pinot cannot find a consensus on validDocs across replicas for the same completed segment. Cluster shape: - 2 controllers - 6 brokers - 8 servers - 4 stateless minions - 3 ZooKeeper nodes - S3-compatible deep store - Kafka ingestion Cluster sizing / capacity numbers: Controllers: - replicas: 2 - per pod: 2 CPU request / 4 CPU limit - per pod memory: 8 GiB request / 8 GiB limit - per pod JVM heap: Xms 4G / Xmx 6G - total controller CPU: 4 requested / 8 limited - total controller memory: 16 GiB requested / 16 GiB limited - total controller heap: 8G Xms / 12G Xmx Brokers: - replicas: 6 - per pod: 1 CPU request / 2 CPU limit - per pod memory: 9 GiB request / 12 GiB limit - per pod JVM heap: Xms 6G / Xmx 6G - total broker CPU: 6 requested / 12 limited - total broker memory: 54 GiB requested / 72 GiB limited - total broker heap: 36G Servers: - replicas: 8 - per pod: 6 CPU request / 8 CPU limit - per pod memory: 32 GiB request / 48 GiB limit - per pod JVM heap: Xms 16G / Xmx 28G - persistent storage per server: 600 GiB - total server CPU: 48 requested / 64 limited - total server memory: 256 GiB requested / 384 GiB limited - total server heap: 128G Xms / 224G Xmx - total server local persistent storage: 4.8 TiB Minions: - replicas: 4 - stateless minions - per pod: 2 CPU request / 4 CPU limit - per pod memory: 8 GiB request / 10 GiB limit - per pod JVM heap: Xms 4G / Xmx 6G - per pod ephemeral storage: 30 GiB request / 100 GiB limit - task executor threads per minion: 4 - total minion CPU: 8 requested / 16 limited - total minion memory: 32 GiB requested / 40 GiB limited - total minion heap: 16G Xms / 24G Xmx - total minion ephemeral storage: 120 GiB requested / 400 GiB limited - maximum theoretical minion task slots: 4 minions * 4 threads = 16 parallel tasks ZooKeeper: - replicas: 3 - persistent storage per pod: 20 GiB - total ZooKeeper storage: 60 GiB ealtime table layout: - most tables use 16 logical partitions - replicasPerPartition: 2 - consuming assignment uses replica-group based partition assignment - roughly 16 partitions * 2 replicas = 32 consuming segment assignments per table - across 8 servers this is roughly 4 consuming assignments per server per table - with 4 similar realtime tables this can be roughly 128 consuming assignments total, or around 16 per server, depending on the current segment state Segment settings: - realtime segment flush threshold size: mostly 250-300 MB - realtime segment flush threshold time: 1 day - realtime segment flush threshold rows: 0 Table patterns: The most stable table is a FULL upsert latest-state table: - dropOutOfOrderRecord=true - validDocIdsType=IN_MEMORY - invalidRecordsThresholdPercent=5 - tableMaxNumTasks=1 - schedule every 10 minutes - latest-state semantics The less stable tables are PARTIAL upsert aggregation/history tables: - INCREMENT metrics - dropOutOfOrderRecord=false - validDocIdsType=IN_MEMORY - tableMaxNumTasks=2 - schedules staggered every 5 minutes across tables - aggregation/history-style primary keys Current task scheduling: - latest-state FULL upsert table: every 10 minutes, tableMaxNumTasks=1 - monthly PARTIAL aggregation table: every 20 minutes, tableMaxNumTasks=2 - daily PARTIAL aggregation table: every 20 minutes, tableMaxNumTasks=2 - history-like PARTIAL table: every 20 minutes, tableMaxNumTasks=2 The schedules are staggered, but close together: - monthly table: minute 0/20/40 - daily table: minute 5/25/45 - history-like table: minute 10/30/50 - latest-state table: every 10 minutes So the configured maximum task count across these tables is roughly: - latest-state table: 1 - monthly aggregation table: 2 - daily aggregation table: 2 - history-like table: 2 - total configured table task capacity: up to 7 UpsertCompactionTasks However, because schedules are frequent and tasks may run longer than the schedule interval, we suspect that task backlog or overlap could still create contention against the 16 available Minion task slots. Questions: 1. What are the typical causes of this error during UpsertCompactionTask? "...java.lang.IllegalStateException: No consensus on validDocs across replicas for segment ... Failing task to avoid replica inconsistency." 2. Which metrics/logs best show whether this is caused by validDocs mismatch, CRC mismatch, Minion resource pressure, S3/deep-store issues, or Helix/segment replacement timing? 3. Does our cluster sizing look sufficient for this workload, or are Minions/Servers/deep store likely under-provisioned? Any recommendations on how to debug and stabilize UpsertCompactionTask in this setup would be very helpful. Thanks in advance for your help!

I have a question about using replica groups in Pinot. Replica groups help control query fanout, but is there any recommendation for how many servers a single tenant should have before it makes sense to enable replica groups? In my experience, when the number of servers is small, using replica groups actually makes queries slower because the load concentrates on fewer servers. Switching back to balanced segment assignment makes queries faster. I’m not sure at what server count (per tenant) it’s recommended to start using replica groups, and how I should properly observe or measure query fanout to decide when it becomes a problem. Any guidance or best practices on this would be appreciated.

I have been looking into what would be needed to run pinot tests in parallel - it's mostly collisions in networking ports,and test files. Concurrent tests may use same ports or write same files. It runs the entire suite in ~30m on my machine. Is there any interest making PRs for those changes? It can get pretty involved.

fyi

How large has a single Pinot table been scaled in production? Anyone running > 150TB in single prod table? Looking for real-world upper bounds on one table (not cluster-wide totals) - specifically total row count, on-disk size, and segment count. Questions: 1. What's the largest single table you run, by rows / bytes / segment count? 2. What became the bottleneck first as segment count grew — ZK metadata + ideal-state size, Helix state transitions, broker routing-table build, controller rebalance duration, server startup/mmap time, or minion throughput? 3. Is there a practical segment-count ceiling per table (and per server) before ops pain starts? 4. At this scale, what segment size are you targeting? We're wondering whether fewer/larger segments is the right trade against query-time pruning. 5. Anyone running >100K segments in one table — what tuning was required to keep it healthy?

Thanks @Xiang Fu, any known min and max safe limits for Djute.maxbuffer, what's the max we can configure for 300k segments, in future it can grow up to 500k segments ? Thread in Slack Conversation

Hi Team Are there any plans to back port bug fixes to older releases as well?

Hey everyone! Sharing a technical article from a project we recently worked on. We explored how to build a hallucination-free Text-to-SQL data agent using MCP, DataHub, Apache Pinot 🍷, and Databricks - with a focus on getting the right data source and generating reliable SQL without fine-tuning a model. Hallucination-free Text-to-SQL: How We Built a Data Agent with MCP, DataHub, Pinot, and Databricks medium.com/@ignitzhjfk/hallucination-free-text-to-sql-how-we-built-a-data-agent-with-mcp-datahub-pinot-and-databricks… Would love to hear your thoughts and feedback, especially from anyone working with AI agents, data platforms, or Text-to-SQL.

Hi All, Is any on eusing apache pinot 1.5.1 version ? How to build source code in local and create tar file. Can anyone help me on this

👋 Hi people. I created a

pinot-gateway

to route and log queries to kafka or database as a proxy between the client and Pinot Broker. github.com/ignitz/pinot-gateway If you wanna learn more about it, can check in this post. medium.com/@ignitzhjfk/pinot-gateway-closing-the-observability-gap-in-apache-pinot-with-a-query-logger-and-an-ai-feedback…

Hey team, 👋 here is the Pinot team at Slack with the question about the safety Upsert table at ~40% Old Gen with 0 old-gen GC We're running Pinot 1.4.0 with a FULL-upsert realtime table (~1.2B PKs, MURMUR3, 2d retention, 24 servers,

-Xmx 38G

,

-XX:InitiatingHeapOccupancyPercent=25

). After tuning (retention reduction, Xmx increase, instance spread), Old Gen sits at 38–44% depending on the PK sawtooth phase. The PK map is 98% of heap (

jmap -histo:live

verified, 206 bytes/entry). We see zero old-gen GC events, 0 query failures, and P50 180ms across 7,200-query benchmarks. However, @Mayank also provided the guideline recommending Old Gen <30% of Xmx for upsert tables or max heap < 70% of Xmx . So we want to double check if one of the following condition a safe steady state for production 1. ~45% Old Gen with 0 old-gen GC 2. max heap < 70% most of time but some seldom random spike to > 80%

hey team, we are running approx 300 servers in a pinot cluster with only 3 zookeepers and controllers. while these dont necessarily need to be scaled up as they operate in leader follower configs, do you see any issue with keeping the control path this barebones while scaling up server count? (its safe to say there is ~300TB of data)

Hey team, we are hosting realtime upsert tables in Pinot 1.4, want to double check if

completed

block shown as below is helpful for prevent segment relocation after completion thanks

"COMPLETED": {
                "tagPoolConfig": {
                    "tag": "general-realtime",
                    "poolBased": true,
                    "numPools": 0
                },
                "replicaGroupPartitionConfig": {
                    "replicaGroupBased": true,
                    "numInstances": 0,
                    "numReplicaGroups": {{ table_attributes.num_replica_groups }},
                    "numInstancesPerReplicaGroup": {{ table_attributes.num_instances_per_replica_group }},
                    "numPartitions": {{ table_attributes.num_segment_partitions }},
                    "numInstancesPerPartition": 1,
                    "minimizeDataMovement": true
                },
                "partitionSelector": "INSTANCE_REPLICA_GROUP_PARTITION_SELECTOR"
            }

Hi all, The Apache Pinot team is pleased to announce Apache Pinot 1.6.0! 🎉 This release includes: • SQL DDL and materialized-view support •

GROUPING SETS

,

ROLLUP

, and

CUBE

• New vector-index backends • Ingestion, query, and reliability improvements Upgrade note: Pinot services now require Java 25+. Java/JDBC clients and explicitly targeted SPI/common artifacts remain Java 11-compatible. Please review the upgrade notes, removals, and SMART aggregation mixed-version restrictions before upgrading. 📦 Downloads 📝 Release notes and upgrade guidance 📚 1.6.0 documentation Thanks to everyone who contributed, tested, and voted for this release!

Hi everyone! We're planning the Apache Pinot 2026 Year in Review, a virtual community call in the second week of December, and we're looking for speakers. We'd love to hear from you if you have: • A customer success story: how Pinot is used at your company • A feature you added to Pinot this year • Your story as a new contributor: how you got started and tips for others who want to contribute Each speaker gets 2–3 slides and 2–3 minutes. If you're interested, please drop me a message!

Hi everyone, I have a question regarding memory sizing for Apache Pinot, specifically the balance between JVM heap, off-heap memory, and total RSS across processes. Our node has 252 GB of physical RAM. The current JVM heap allocations are: • Pinot Server: 52 GB (

-Xmx52g

) • Pinot Minion: 32 GB (

-Xmx32g

) • Pinot Broker: 24 GB (

-Xmx24g

) • ZooKeeper: 1 GB (

-Xmx1g

) We are considering increasing only the Pinot Server heap from 52 GB to 132 GB (

-Xmx132g

). The heap allocations for the Minion, Broker, and ZooKeeper will remain unchanged. My main concern is not just heap usage, but the total RSS (physical memory consumption) of all processes, since RSS includes heap, off-heap memory, memory-mapped segments, native allocations, etc. My questions: 1. How much physical RAM should ideally remain available for off-heap memory and OS filesystem cache after accounting for all JVM heaps? 2. Is there a recommended heap-to-off-heap ratio for Pinot Servers handling heavy query workloads? 3. How should we account for the combined RSS of Server, Minion, Broker, and ZooKeeper when deciding the Server’s maximum heap? 4. Would increasing the Server heap from 52 GB to 132 GB improve query performance, or could it negatively affect performance by reducing memory available for mmap pages and filesystem caching? 5. Are there any real-world production recommendations for heap sizing on a 252 GB RAM node? Would appreciate suggestions based on production experience.