Hi all! I've proposed a new feature for Pinot: lazy segment loading for OFFLINE tables, servers register metadata-only stubs on segment assignment and materialize them from the deep store on first query, with idle eviction back to stubs. Disk/memory then scale with the working set instead of total assigned data. Server-side only, off by default, opt-in per table. Design + known limitations: github.com/apache/pinot/issues/18986 PR: github.com/apache/pinot/pull/18987 I've been running this in (Pinot 1.5.0 base, MinIO deep store, incl. a 10 GB / 135M-row test). Feedback very welcome!

This looks really cool! I have two broad questions: 1. how would a user know if this would be beneficial to them? i don't think we have any pinot metrics that say "40% of the segments are unused" 2. What would happen if the lazily fetched segments cause the server to run out of disk space?

Thanks Tim, great questions! 1. How would a user know it'd benefit them? You're right that there's no "X% of segments are unused" metric in Pinot today — that's a gap. Two answers: - Heuristic (today): the strongest signal is retention window vs. query window on time-partitioned tables. If you retain 1 year but dashboards only ever hit the last 30 days, ~92% of your segments are cold — you don't need a metric to know that, it falls out of the table config and query patterns. - Measured (with this feature): the implementation already tracks per-segment last-access time to drive eviction, so it naturally produces exactly that metric — materialized vs. stubbed segment counts show your real working set. A nice follow-up would be exposing a last-access timestamp / "cold segment ratio" gauge in stock Pinot even with lazy loading off, so users can evaluate before enabling. Happy to add that. 2. Running out of disk from lazily fetched segments? The TTL sweeper is the main pressure valve — idle segments revert to stubs and local files get deleted, so disk usage tracks the working set, not total volume. If a materialization does fail (e.g. disk full mid-download), the stub is retained, the query reports those segments as unavailable rather than the server crashing, and the next query retries. What's not in the current PR is proactive disk-pressure-based eviction — e.g. a disk budget with LRU eviction at a high-watermark instead of waiting for the TTL. That's the natural next step and I'd love input on the right shape for it (per-table budget vs. instance-wide, etc.).