Hello Pinot Community, This is a call for a vote to release Apache Pinot 1.4.0. I also sent this to the dev@pinot in case you didn’t see it. • The release candidate: https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-1.4.0-rc0/ • Git tag for this release: https://github.com/apache/pinot/releases/tag/release-1.4.0-rc0 • Git hash for this release: 3a56c9db545e3b403863d8319055ea9be8eae771 • The artifacts have been signed with the key: F0CC31868DCDC6E3E8D98905205333AA7B9B9619, which can be found in the KEYS file: dist/release/pinot/KEYS • Release Notes: https://github.com/apache/pinot/releases/tag/release-1.4.0-rc0 • Staging repository: https://repository.apache.org/content/repositories/orgapachepinot-1070/ • Documentation on verifying a release candidate: https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate The vote will be open for at least 72 hours or until a necessary number of votes are reached. Please vote accordingly, [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason Thanks, Apache Pinot Team cc: @Mayank @Xiang Fu

I’m currently running a Pinot cluster with Kafka, server, broker, Zookeeper, and controller. I noticed that the setup is consuming around 8GB out of 16gb of memory at idle state. Is this expected for such a deployment, or should I optimize something? Also, I’d love to get your insights on best practices for Pinot deployment. Thanks for your time! Best regards, Sathwik

Hi Team, I'm facing issues with implementing pagination using cursor approach . Any guidance is appreciated . Followed the document still not able to get paginated response .

Hi All, Is it possible to create multiple tenants in an Apache Pinot cluster that has only one server and one broker?

I successfully set up multiple tenants using a single server and broker, but I am sure about the method is used to create multiple tenants is correct or not. Could anyone please verify it and give me suggestions...Here’s how I did it: First, I used the API endpoint

PUT /instances/{instanceName}/updateTags

to add the necessary tags to both the server and the broker. After that, I included the corresponding tenant names in the table configuration for each specific table. I also have a question: when does the concept of tenants actually come into play in practice?

👋 I have a PR here to fix the HttpsSegmentFetcher for our TLS setup. We found that peer downloading was not working because it was relying on old configs which we no longer use. With this change we were able to peer fetch segments with https again. Please take a look when you have time https://github.com/apache/pinot/pull/17315 thanks

https://github.com/apache/pinot/pull/17177 @Kishore G can you help approve the PR is it looks good to you? All comments have been addressed, thanks!

Hi, I want to start a discussion about removing

createInvertedIndexDuringSegmentGeneration

field so that inverted index behavior aligns with all other index types. Currently inverted index is not created by default (see this section) during segment creation, but all other indices are created. The proposal is to remove this flag and always create inverted index during segment creation as well. The side effect is larger deep store footprint, but reduced segment load time and consistent behavior across all indices. Please chime in if you have concern on this proposal.

Hi, in our recent observations with OOM query killing, we've found some missing instrumentations/bugs in the implementation specifically with Lucene indexes. I have an issue open with some details, and prospective fixes too, can someone familiar with OOM protection code take a prelim look at the github issue and see if this makes sense - https://github.com/apache/pinot/issues/17877

We'll be starting the Apache Pinot 1.5.0 release process soon. The base commit will be frozen later this week and after some extensive testing and certification that is expected to take ~2 weeks, we will start the actual release process which should take about a week or two more. You can join #C0ALUTY8C3G if you're interested in tracking the progress of the release. We'll be using that channel for all release related coordination henceforth. cc - @Mayank @Xiang Fu @Jackie

Hello, I would like to propose extending the existing query killing framework with some proactive query killing in Pinot (cost or pattern based), that will be helpful for a lot of cases. Please see the motivation and high level design and let me know if any questions/concerns. Will share detailed design in some time if no issues. CC @Jackie

Hi, the recommended deployment order for pinot is controller -> broker -> server -> minion, and I see on the docs that it's a recommendation rather than hard requirement. If I'm running pinot across 3 kubernetes clusters (each cluster with their own controllers/brokers/servers/minions), is there a requirement that the controller needs to be deployed across all 3 clusters, before any other component? meaning controller in first cluster, then second, then third clusters, and then broker/server/minion in other clusters? or can i do controller -> broker -> server -> minion in first cluster, then second, and then third?

Hey, I noticed that Add broker endpoint to validate parsing of queries #17615 has three PRs open for it: • Adds a broker endpoint to validate query parsing without further validation/execution #17618 • Fix #17615: Add broker endpoint to parse SQL queries without execution #17661 • Add POST /query/sql/validateSyntax broker endpoint #18281 Can we get a review on one of these PRs?

edit: closing while I look at https://github.com/apache/pinot/pull/18530 👋 Hello, I have a small update here to make the segment upload timeout configurable on the PushJobSpec. Please take a look when you have time 😄 https://github.com/apache/pinot/pull/18617

Hi , I came across this https://github.com/apache/pinot/issues/17398 , i see it has been open from December , can i pick this up? is there a reason why it has been kept aside?

hey, i came across this issue while working with Pinot and hdfs can someone who is familiar with the hdfs plugin take a look at this issue: https://github.com/apache/pinot/issues/18499 and see if it makes sense to them

also, does pinot have pre-existing support for an event listener when a segment is pushed to the deep store? it would be a good-to-have feature, and would allow users to maybe trigger some outside jobs/actions (for example replicating the segment pushed to deep store) when a segment is committed to the deep store.

Hey @Jia Guo and @Sidd, do you have any concerns on https://github.com/apache/pinot/pull/18286? I have a follow up PR that allows dynamic tuning of the adaptive routing parameters (a huge time saver for tuning adaptive routing on our clusters), and I'd love to get it merged.

✏️ Would love to get some reviews on this

ArrowColumnReader

implementation (context in this issue) https://github.com/apache/pinot/pull/18632 https://github.com/apache/pinot/pull/18638

hey, during recent work with Pinot I have felt that the below feature could be a good addition in case a user wanted to gather segment metadata in near realtime. Would be great if we could discuss on whether this is something that would be a good-to-have feature in Pinot and if it makes sense (if it does I will come up with a plan) https://github.com/apache/pinot/issues/18712

I recently opened https://github.com/apache/pinot/issues/18740 and https://github.com/apache/pinot/pull/18741, adding the scaffolding and the first use of Cost-Based Optimizer (CBO) into MSE. Feel free to take a look and review it!

Hi, could someone take a look at this Bug report ? We've seen this a few times in our clusters where an upsert table consumption of a partition is stuck on the semaphore unless manually mitigated. And we've RC-ed it to this issue.

Hi, One more small bug fix in Lucene text index querying when

reuseMutableIndex

is used. This one's simple. Can someone review and merge ? thanks CC @Mayank @Jackie

anyone able to review query log redaction? github.com/apache/pinot/pull/18604

i see combatibiltiy checks are failing in my pr github.com/apache/pinot/pull/18751 but none of these are my changes , how can i resolve them?

Hi, can someone please take a look at these two bug fixes. github.com/apache/pinot/pull/18733 github.com/apache/pinot/pull/18813 thanks

hi can someone take a look at this issue github.com/apache/pinot/issues/18836 is it something we want to fix?

Hi all! I've proposed a new feature for Pinot: lazy segment loading for OFFLINE tables, servers register metadata-only stubs on segment assignment and materialize them from the deep store on first query, with idle eviction back to stubs. Disk/memory then scale with the working set instead of total assigned data. Server-side only, off by default, opt-in per table. Design + known limitations: github.com/apache/pinot/issues/18986 PR: github.com/apache/pinot/pull/18987 I've been running this in (Pinot 1.5.0 base, MinIO deep store, incl. a 10 GB / 135M-row test). Feedback very welcome!

👋 I wanted to raise a topic that I think is worth discussing as Pinot adoption grows: ZooKeeper scaling under very high segment counts. We’re running a deployment with some tables pushing past 500,000 segments. At that scale we’re starting to see real pressure on ZooKeeper. IdealState update times getting into the tens of seconds, periodic tasks that do full O(N) ZNode reads on every cycle, and the general concern that ZK isn’t really designed to be a data store at this scale (the commonly cited guidance is to keep ZNode counts well under 100k). Some simple API endpoints like listing all segments in the table start to breakdown as well. Even approaches like segment rollups or logical tables help operationally but don’t fundamentally reduce ZNode count when you’re in the multi-hundreds-of-thousands range. Our infra teams are already a bit scared at the znode count and jute max buffer size we are approaching 😟 Have some more thoughts in thread 🧵

I have a few changes to support MSE adaptive routing. • Add latency tracking for MSE adaptive routing • Make adaptive routing variables runtime adjustable • [#12507] Implement adaptive pool selection for StrictReplicaGroupInstanceSelector - fixes correctness bugs with SSE adaptive routing + upsert tables @Jia Guo / @Sidd / @Vivek Iyer Vaidyanathan / @Xiang Fu can you TAL at these? They've been deployed on Stripe's clusters for a few weeks / months and working well.