We'll be starting the Apache Pinot 1.5.0 release process soon. The base commit will be frozen later this week and after some extensive testing and certification that is expected to take ~2 weeks, we will start the actual release process which should take about a week or two more. You can join #C0ALUTY8C3G if you're interested in tracking the progress of the release. We'll be using that channel for all release related coordination henceforth. cc - @Mayank @Xiang Fu @Jackie

Hello, I would like to propose extending the existing query killing framework with some proactive query killing in Pinot (cost or pattern based), that will be helpful for a lot of cases. Please see the motivation and high level design and let me know if any questions/concerns. Will share detailed design in some time if no issues. CC @Jackie

Hi, the recommended deployment order for pinot is controller -> broker -> server -> minion, and I see on the docs that it's a recommendation rather than hard requirement. If I'm running pinot across 3 kubernetes clusters (each cluster with their own controllers/brokers/servers/minions), is there a requirement that the controller needs to be deployed across all 3 clusters, before any other component? meaning controller in first cluster, then second, then third clusters, and then broker/server/minion in other clusters? or can i do controller -> broker -> server -> minion in first cluster, then second, and then third?

Hey, I noticed that Add broker endpoint to validate parsing of queries #17615 has three PRs open for it: • Adds a broker endpoint to validate query parsing without further validation/execution #17618 • Fix #17615: Add broker endpoint to parse SQL queries without execution #17661 • Add POST /query/sql/validateSyntax broker endpoint #18281 Can we get a review on one of these PRs?

edit: closing while I look at https://github.com/apache/pinot/pull/18530 👋 Hello, I have a small update here to make the segment upload timeout configurable on the PushJobSpec. Please take a look when you have time 😄 https://github.com/apache/pinot/pull/18617

Hi , I came across this https://github.com/apache/pinot/issues/17398 , i see it has been open from December , can i pick this up? is there a reason why it has been kept aside?

hey, i came across this issue while working with Pinot and hdfs can someone who is familiar with the hdfs plugin take a look at this issue: https://github.com/apache/pinot/issues/18499 and see if it makes sense to them

also, does pinot have pre-existing support for an event listener when a segment is pushed to the deep store? it would be a good-to-have feature, and would allow users to maybe trigger some outside jobs/actions (for example replicating the segment pushed to deep store) when a segment is committed to the deep store.

Hey @Jia Guo and @Sidd, do you have any concerns on https://github.com/apache/pinot/pull/18286? I have a follow up PR that allows dynamic tuning of the adaptive routing parameters (a huge time saver for tuning adaptive routing on our clusters), and I'd love to get it merged.

✏️ Would love to get some reviews on this

ArrowColumnReader

implementation (context in this issue) https://github.com/apache/pinot/pull/18632 https://github.com/apache/pinot/pull/18638

hey, during recent work with Pinot I have felt that the below feature could be a good addition in case a user wanted to gather segment metadata in near realtime. Would be great if we could discuss on whether this is something that would be a good-to-have feature in Pinot and if it makes sense (if it does I will come up with a plan) https://github.com/apache/pinot/issues/18712

I recently opened https://github.com/apache/pinot/issues/18740 and https://github.com/apache/pinot/pull/18741, adding the scaffolding and the first use of Cost-Based Optimizer (CBO) into MSE. Feel free to take a look and review it!

Hi, could someone take a look at this Bug report ? We've seen this a few times in our clusters where an upsert table consumption of a partition is stuck on the semaphore unless manually mitigated. And we've RC-ed it to this issue.

Hi, One more small bug fix in Lucene text index querying when

reuseMutableIndex

is used. This one's simple. Can someone review and merge ? thanks CC @Mayank @Jackie

anyone able to review query log redaction? github.com/apache/pinot/pull/18604

i see combatibiltiy checks are failing in my pr github.com/apache/pinot/pull/18751 but none of these are my changes , how can i resolve them?

Hi, can someone please take a look at these two bug fixes. github.com/apache/pinot/pull/18733 github.com/apache/pinot/pull/18813 thanks

hi can someone take a look at this issue github.com/apache/pinot/issues/18836 is it something we want to fix?

Hi all! I've proposed a new feature for Pinot: lazy segment loading for OFFLINE tables, servers register metadata-only stubs on segment assignment and materialize them from the deep store on first query, with idle eviction back to stubs. Disk/memory then scale with the working set instead of total assigned data. Server-side only, off by default, opt-in per table. Design + known limitations: github.com/apache/pinot/issues/18986 PR: github.com/apache/pinot/pull/18987 I've been running this in (Pinot 1.5.0 base, MinIO deep store, incl. a 10 GB / 135M-row test). Feedback very welcome!

👋 I wanted to raise a topic that I think is worth discussing as Pinot adoption grows: ZooKeeper scaling under very high segment counts. We’re running a deployment with some tables pushing past 500,000 segments. At that scale we’re starting to see real pressure on ZooKeeper. IdealState update times getting into the tens of seconds, periodic tasks that do full O(N) ZNode reads on every cycle, and the general concern that ZK isn’t really designed to be a data store at this scale (the commonly cited guidance is to keep ZNode counts well under 100k). Some simple API endpoints like listing all segments in the table start to breakdown as well. Even approaches like segment rollups or logical tables help operationally but don’t fundamentally reduce ZNode count when you’re in the multi-hundreds-of-thousands range. Our infra teams are already a bit scared at the znode count and jute max buffer size we are approaching 😟 Have some more thoughts in thread 🧵

I have a few changes to support MSE adaptive routing. • Add latency tracking for MSE adaptive routing • Make adaptive routing variables runtime adjustable • [#12507] Implement adaptive pool selection for StrictReplicaGroupInstanceSelector - fixes correctness bugs with SSE adaptive routing + upsert tables @Jia Guo / @Sidd / @Vivek Iyer Vaidyanathan / @Xiang Fu can you TAL at these? They've been deployed on Stripe's clusters for a few weeks / months and working well.

Hi Team, can someone help review this Bug fix for

skipUnavailableServers

not working as intended in case of channel-inactive errors. thanks

hi can someone pls review this PR for excluding out of retention records from the result set (issue is linked to the PR) thanks

Hi team, we plan to release 1.6.0 and the official cutoff date is Aug 7. Please plan your dev work and communicate in #C0BLKT1BB1N channel if you want your work to be included in the 1.6.0 release. Thanks!!

Hi team, can someone please help review this PR Add controller support for rejecting out-of-retention segment uploads

is this a known issue with Pinot? github.com/apache/pinot/issues/18902#…

Hello 👋 I just opened a small but important fix for an S3 data-loss race condition:

S3PinotFS#copyDir

was deleting the destination object before copying the new one in, meaning a crash between those two operations would permanently lose the segment in S3 with no recovery path. S3 CopyObject is already an atomic overwrite (no delete needed), and GCS has handled this correctly all along. This one-line fix brings S3 in line. Affects METADATA push, LLC segment commit/repair, segment deletion staging, and batch ingestion jobs on S3 deep stores. PR: github.com/apache/pinot/pull/19337 Please take a look when you have time

Hi folks, we've encountered some race conditions that can introduce CRC mismatch between segment replica CRCs e.g., time column null being overridden to System.now(). This creates risk of losing segments that lack a deep store copy. To catch any other such edge case, we wanted to add more observability for CRC mismatch replacements. I'd appreciate if someone can help review this change github.com/apache/pinot/pull/19450.

hi team, please have a look at the first cut of PEP for system tables in Pinot. Doc: docs.google.com/document/d/1l81rQ4VVj9M…/edit?usp=sharing the issues are linked in the doc itself. the current state of the PEP serves as an initial design doc for architectural discussion and review. cc: @Jackie

Hi Team, Can someone review this PR : github.com/apache/pinot/pull/19529 Context: Pinot currently supports changing decoder of a live table and allows new consuming segments to pick the new decoder up. But there's no observability or metadata stored anywhere what decoder is currently being used for a segment or what decoder was used to create the segment. The PR fixes that. thanks