Hi Pinot community, We are evaluating Apache Pinot as a low-latency serving layer as part of our migration from Snowflake to an open lakehouse architecture, and I would appreciate some guidance on whether our proposed architecture aligns with Pinot best practices. Our current stack is: Trino + Apache Iceberg + Azure ADLS Gen2/HNS + Snowflake Open Catalog/Polaris + Spark + Flink + Red panda + Azure AKS*.* The use case is building/sensor analytics exposed through an application dashboard. We need to serve both recent/raw sensor telemetry and aggregated KPIs. Spark/Flink perform transformations and aggregations, with Iceberg on ADLS Gen2 acting as the durable analytical/system-of-record layer. One of our current challenges is KPI latency through the Trino → Iceberg → ADLS path. We also have an Iceberg small-file accumulation issue, so we want to avoid introducing Pinot in a way that creates additional heavy reads against the same ADLS storage. The architecture we are considering is: 1. Redpanda → Pinot for recent/realtime telemetry 2. Redpanda → Spark/Flink → Iceberg/ADLS for the durable lakehouse 3. Spark/Flink computes aggregates once and publishes the result to: ◦ Iceberg Gold for durable/historical storage ◦ Redpanda aggregate topic → Pinot for application KPI serving The application would then route: KPI / recent telemetry → Pinot Historical scans / large exports / ad-hoc SQL → Trino → Iceberg → ADLS We would use Iceberg/ADLS → Spark → Pinot primarily for initial historical loads, backfills or corrections, rather than continuously scanning Iceberg to refresh Pinot. Could the community please help us validate the following? 1. Is this a recommended pattern for using Pinot as a dedicated application-facing serving layer while Iceberg/ADLS remains the system of record? 2. For realtime ingestion, are there any known considerations when using Redpanda through its Kafka-compatible API as the Pinot stream source, particularly with SASL/TLS and consumer/offset behavior? 3. For Spark/Flink-produced aggregates, is publishing the aggregate once to Iceberg and an aggregate Kafka/Redpanda topic → Pinot preferable to having another Spark job re-read Iceberg to populate Pinot? 4. Is Azure ADLS Gen2 with HNS a well-supported production option for Pinot deep storage? Are there any specific HNS/ABFS/authentication limitations or configuration recommendations we should be aware of? 5. For Pinot on Azure AKS, would the recommended pattern be persistent/block storage such as Azure Disk for Pinot Server serving storage, with ADLS used as deep storage? 6. Does Pinot need any direct integration with Iceberg REST catalogs such as Snowflake Open Catalog/Polaris, or is it preferable for Pinot to remain completely independent of the Iceberg catalog/metadata path? 7. Given that we currently have an Iceberg small-file problem, is our assumption correct that Pinot can isolate the application KPI SLA from the Iceberg/ADLS path, while the underlying Iceberg compaction/file-layout issue still needs to be resolved independently? 8. For sensor dashboards containing recent raw telemetry plus hourly/daily/monthly aggregates, would you recommend purpose-built Pinot tables with different retention/indexing strategies rather than replicating the complete Iceberg lake into Pinot? Our main objective is to prevent customer-facing KPI latency from being dependent on Iceberg file layout, ADLS latency and Trino concurrency, without introducing additional contention for Spark/Flink/ADLS. Any recommendations or experiences with similar Pinot + Azure + Kafka/Red panda + Spark/Flink + Iceberg architecture would be greatly appreciated. Thanks!

1. Yes. You can also ingest both stream as well as lake data into pinot and serve them unified. 2. Red panda with Kafka compatible API should work 3. Publish to both will make it simpler for you to not have a separate batch ingestion pipeline for Pinot. 4. ADLS Gen2 is supported 5. Pinot server instances should have local or network attached disks (EBS equivalent). Mounting ADLS will have performance issues. 6. No need for integration with Iceberg REST catalogs. 7. If you go with 3 above, then file count in Iceberg doesn't matter. 8. Single table can have multiple rollups pre-materialized (with ST index), or materialize at read time.