I have a use case where I'm ingesting data from multiple sources into Pinot. After ingestion, I perform various aggregations. My goal is to retain a raw, detailed table and then run rollup jobs to create multiple aggregated views (e.g., hourly, daily) in separate Pinot tables.
However, as mentioned earlier, Pinot’s built-in Minion Merge Rollup task doesn’t support writing rolled-up segments into a different table — it only works within the same table.
Given that limitation, what’s the recommended approach for this kind of setup?
Should I:
1. Combine all data from different sources into a single enriched Kafka topic and ingest that into Pinot?
2. Ingest data from each source into separate tables and perform joins in Pinot?
3. Or take a hybrid approach — keep the raw combined data for ad hoc queries and use external rollup pipelines (or custom Minion tasks) to populate aggregated tables?
Would love to hear how others are handling similar patterns.