Hello Team, Is it possible to configure a Pinot Mi...
# general
a
Hello Team, Is it possible to configure a Pinot Minion Merge Rollup task to read segments from one source table and write the rolled-up data into a different destination table?
m
Not at the moment, because minion tasks are associated with table. Having said that, once a segment is created, it can be pushed to any number of tables
a
I have a use case where I'm ingesting data from multiple sources into Pinot. After ingestion, I perform various aggregations. My goal is to retain a raw, detailed table and then run rollup jobs to create multiple aggregated views (e.g., hourly, daily) in separate Pinot tables. However, as mentioned earlier, Pinot’s built-in Minion Merge Rollup task doesn’t support writing rolled-up segments into a different table — it only works within the same table. Given that limitation, what’s the recommended approach for this kind of setup? Should I: 1. Combine all data from different sources into a single enriched Kafka topic and ingest that into Pinot? 2. Ingest data from each source into separate tables and perform joins in Pinot? 3. Or take a hybrid approach — keep the raw combined data for ad hoc queries and use external rollup pipelines (or custom Minion tasks) to populate aggregated tables? Would love to hear how others are handling similar patterns.
m
Why do you need to maintain the different rollups? Is it because that’s what you did in your legacy system? If so, I’d recommend evaluating if that is really needed when using Pinot. Having said that, there is StarTree index you can use to maintain different roll ups in the same table
a
yes StarTree index would help here. I can keep raw table as it is — and still benefit from StarTree indexing to get my aggregates. Will Pinot automatically reprocess existing segments with the new StarTree config?
m
StarTree indexing is like any other indexing, you don’t necessarily need to have a minion job to rollup, you can create them as part of segment build
a
If StarTree indexing already handles roll-ups efficiently, in what scenarios does using the Minion Merge Rollup task still make sense?
m
ST index is a lot more than rollup