https://apache-pinot.slack.com/archives/C011C9JHN7R/p1741911243212059 cross-posting for visibility 😆

Created a PR https://github.com/apache/pinot/pull/15377 May someone have a look?

Hey all! The Apache Pinot Contributor call recording is available here #C08H9R5MHTN

Hi, while generating response to a SQL request, which layer in codebase writes to the response?

Watch on-demand: •

Apache Pinot Contributor Call Kickoff | April 3rd, 2025▾

Then tell us what you think: • Apache Pinot Contributor Call Survey Slack Conversation

Hi all, we have started facing node volume affinity issue in our pinot pods after we did an AMI version update to our EKS node groups on which our Pinot is hosted. We had only a single pod(for controller and zookeeper) and since PVCs are AZ bound and the EKS on AMI update changed the AZ in which the node is present we faced this issue. We are planning on having one nodegroup per every AZ(total 3) we have. 3 node groups of controller and zookeeper. Are there any best practices we can follow to avoid this?

Hi All , as part of vulnerability fix avro version didn't changed from 1.11.3 to 1.11.4 . In pinot pom.xml we are seeing that version is updated to 1.11.4 but in the dependencies jar we are not seeing the updated version . could you please let us know on this

Some basic questions. When an index is created, like for null values,what component decides that a given value corresponds to null value. For eg, for a float column, -infinity value is mapped to null. Same for NaN and +infinity.

Basically I am looking into population of null bit maps.

I see few files getting created in the process...columns.psf, floatfield.bitmap.nullvalue.

But couldnt trace where exactly population happens.

@Xiang Fu What’s the difference between Minion and Minion Stateless?

Hi folks, I saw there was a recent change aimed at detecting backwards incompat api changes for pinot-spi - I found this makes it pretty difficult to add more args for index config constructors. Eg., trying to add a new arg for JsonIndexConfig, we now see the error:

Error:  Failed to execute goal com.github.siom79.japicmp:japicmp-maven-plugin:0.23.1:cmp (default) on project pinot-spi: There is at least one incompatibility: org.apache.pinot.spi.config.table.JsonIndexConfig.JsonIndexConfig(java.lang.Boolean,int,boolean,boolean,java.util.Set,java.util.Set,java.util.Set,java.util.Set,int,boolean):CONSTRUCTOR_REMOVED -> [Help 1]

Any concerns with excluding some config classes (or parent config package) from this check? or should we be creating a new constructor for every config addition going forward?

Apache Pinot + Azure Event Hub Integration: Metadata Fetch Timeout Hey team, I’m integrating Apache Pinot (Dockerized) with Azure Event Hub (Kafka-compatible endpoint) for real-time ingestion. The connection works perfectly from a standalone Python Kafka client using confluent_kafka, but Pinot fails with: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata ✅ Confirmed Working: Pinot Server can curl and TLS handshake with icpeventhubns.servicebus.windows.net:9093 Python consumer successfully consumes data from ICPtrafficdataeventhub topic Using correct sasl.jaas.config format with $ConnectionString and EntityPath streamConfigs use stream.kafka.consumer.prop.* correctly ❌ Still Seeing: Pinot throws a metadata fetch timeout when creating the real-time table, even though connectivity and credentials are confirmed. Stream Config: For your reference ---------------------------------------- "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "icptrafficdataeventhub", "stream.kafka.broker.list": "icpeventhubns.servicebus.windows.net:9093", "stream.kafka.consumer.type": "simple", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.prop.bootstrap.servers": "XXXns.servicebus.windows.net:9093", "stream.kafka.consumer.prop.security.protocol": "SASL_SSL", "stream.kafka.consumer.prop.sasl.mechanism": "PLAIN", "stream.kafka.consumer.prop.ssl.endpoint.identification.algorithm": "", "stream.kafka.consumer.prop.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\\\"$ConnectionString\\\" password=\\\"Endpoint=sb://XXXns.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=My acess key=;EntityPath=icptrafficdataeventhub\\\";", "realtime.segment.flush.threshold.rows": "50000", "realtime.segment.flush.threshold.time": "24h" } #C013WKLT5T7 #C03N1JNHXLY

Hi All , Unable to connect to SSL Enabled Zookeeper getting the following error 2025/04/24 095621.256 WARN [ClientCnxnSocketNetty] [zkNetty-EpollEventLoopGroup-3-1] future isn't success. io.netty.channel.StacklessClosedChannelException: null at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] 2025/04/24 095622.259 INFO [ClientCnxn] [main-SendThread(localhost:2191)] Opening socket connection to server localhost/127.0.0.1:2191. 2025/04/24 095622.260 INFO [ClientCnxn] [main-SendThread(localhost:2191)] SASL config status: Will not attempt to authenticate using SASL (unknown error) 2025/04/24 095622.262 WARN [ChannelInitializer] [zkNetty-EpollEventLoopGroup-3-1] Failed to initialize a channel. Closing: [id: 0xd0530de0] org.apache.zookeeper.common.X509Exception$TrustManagerException: java.lang.IllegalArgumentException: Unable to auto-detect store file type from file name: /nfmt/instance/certificates/External/truststore.ks at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:612) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.common.ClientX509Util.getTrustManager(ClientX509Util.java:198) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.common.ClientX509Util.createNettySslContextForClient(ClientX509Util.java:77) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initSSL(ClientCnxnSocketNetty.java:449) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:438) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:424) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1130) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:558) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:45) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1410) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1064) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:599) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:513) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:485) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at java.base/java.lang.Thread.run(Thread.java:840) [?:?] Caused by: java.lang.IllegalArgumentException: Unable to auto-detect store file type from file name: /nfmt/instance/certificates/External/truststore.ks

is there any way secure the zookeeper logs with ssl

please guide us to solve this where we are unable to make zookeeper as ssl and zookeeper data logs is storing without encryption

Hi all, I am using pinot-offline ingestion to upload data from Hdfs Cluster A into Pinot backed by Hdfs cluster B using spark. The hadoop config is read from the cluster on which the spark job is running. Is there a way to pass Hdfs config of both the clusters together... Or any other approach that you suggest??

Hi @Jackie , @Xiang Fu - Kindly review this : https://github.com/apache/pinot/pull/15844 https://docs.google.com/document/d/1M5OLUkW0eqn6ZnRYH1gg7lh7VrrJyu43gubix8xHm-s/edit?tab=t.0

A quick question,

cluster config setting will take priority or client setting will take priority in pinot ?

i.e: am setting pinot.broker.timeoutMs = 60000 in broker and server config file itself, but if some clients want to override this setting, can they set at client level a new setting like pinot.broker.timeoutMs = 1200000 and get the query executed ?

Created a PR https://github.com/apache/pinot/pull/16038/files

Hi all, I was setting up SSL on controller UI using below configs, is there any way to encrypt the password in controller conf file

# SSL Enablement
controller.tls.keystore.path=/etc/security/certificates/keystore.jks
controller.tls.keystore.password=Hadoop@123
controller.tls.truststore.path=/etc/security/certificates/truststore.jks
controller.tls.truststore.password=Hadoop@123
controller.access.protocols=https
controller.access.protocols.https.port=9443

Hi all , we are seeing this Added support to pause and resume ingestion based on resource utilization (Added support to pause and resume ingestion based on resource utilization by rajagopr · Pull Request #15008 · apache/pinot · GitHub) got merged on feb 17 and we are seeing the code is not there in the 1.3.0 source code . which adds the resource utilization code . could you please let us know on this which apache pinot version will get added

Hi Everyone, We are working on Pinot 1.4 release! I am planning to cut the Pinot 1.4 Release from HEAD on July 7th. All the commits merged into master branch before EOD on July 6 will be part of the release. Thanks cc: @Mayank @Xiang Fu @Ting Chen @Yupeng Fu

hi @Jackie @Xiang Fu requesting a PR review for https://github.com/pinot-contrib/pinot-docs/pull/434 This is a documentation PR for UpsertCompactMergeTask (Small Segment merger task).

Heads up, this one bit us today, the fix in the comments worked for us though. https://github.com/apache/pinot/issues/16283

Apache Pinot Contributor Call #3, Tue, Aug 5, 2025, 8:30 AM | Meetup Join us for our monthly Contributor Office Hours - a casual, open session to: ✅ Present your pull requests, proposals, or ideas ✅ Ask questions and get live feedback ✅ Lear

Developed CopyOnWriteRoaringBitmap working with Professor Lemire : https://github.com/RoaringBitmap/RoaringBitmap/pull/780 Cc : @Jackie

Hey @Xiang Fu @Abhishek Sharma , Good day! We were evaluating offline table segment uploads using Spark SegmentTarPush and encountered an issue mentioned here: #16481 I’ve raised a PR for the same. After these changes, we ran it again and it worked as expected. I’ve included detailed steps to reproduce the issue and the fix. I would appreciate it if you could take a look and review this PR: #16482 Let me know if I need to make any other changes. Happy to contribute! Thanks!

Hi all, 📢 RFC Announcement: Broker and Server Segment Query Cache for Apache Pinot Under the guidance of Pinot PMC Xiangfu, we’re proposing a two-layer query cache for Apache Pinot: 1️⃣ Broker Result Cache – caches fully merged query results. 2️⃣ Server Segment Result Cache – caches per-segment partial results (aggregations, group-bys, selections). By storing reusable results in Query Cache, this feature can cut redundant computation and scanning, significantly reducing latency and cost—especially for high-concurrency workloads on large datasets. 📄 Pinot Issue: https://github.com/apache/pinot/issues/16584 📝 Design Doc: https://docs.google.com/document/d/1qUjLEJnhODD3oJrOP_Q15JP-Tb4RkLLNyDHYaWcztMg/edit?usp=sharing 💬 Discussion Channel: https://app.slack.com/huddle/TDR79RQN8/C09A97SDMTN If you’re interested, please comment in the design doc or join the discussion channel.