This message was deleted.
# generals
Slackbot
06/20/2023, 3:56 PMThis message was deleted.
s
Sergio Ferragut
06/20/2023, 6:49 PMYou need to use range, single_dim or hash partitioning to achieve perfect rollup. If you use MSQ SQL-Based Ingestion you can do range partitioning with a REPLACE statement that does aggregation with the SELECT/GROUP BY and the PARTITION BY & CLUSTERED BY clauses to specify time partitioning and secondary partitioning respectively.
👍 1
s
Suneet Saldanha
06/20/2023, 7:43 PMIn this case, if you are using an index_parallel spec, look for the config
forceGuaranteedRolluptuningConfig👍 1
Suneet Saldanha
06/20/2023, 7:44 PMAs Sergio suggested, +1 for using MSQ if you are on a Druid version that supports it. It's much nicer, and there's a lot less tuning like this
👍 1
c
Caner Kürtür
06/21/2023, 10:58 AMThank you for replies! @Sergio Ferragut @Suneet Saldanha
so in my spec , I have "rollup": true, in "granularitySpec": { section , and "forceGuaranteedRollup": true in tuningConfig as you said
Caner Kürtür
06/21/2023, 10:59 AMwhat is the difference between those? I for sure want to enable rollups during ingestion time when creating segments, because we have some metric columns also specified in metricSpec, counts, doubleSum etc.
Caner Kürtür
06/21/2023, 11:00 AMwould it be okay if I set "forceGuaranteedRollup": to false and keep the rollup=true in granularitySpec? would it still do the job for me and rollup during ingestion?
Caner Kürtür
06/21/2023, 11:02 AMwe are feeding our datasource via kafka supervisor and it is set to fully compacted. after I run index_parallel, it would then do the job of ""forceGuaranteedRollup":true" yeah?
Caner Kürtür
06/21/2023, 11:05 AMthis is how my granularitySpec looks like in index_parallel
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "DAY",
"queryGranularity": "HOUR",
"rollup": true,
"intervals": ["2023-06-14T000000.000Z/2023-06-15T235959.999Z"]
},
s
Suneet Saldanha
06/21/2023, 6:17 PMwould it be okay if I set "forceGuaranteedRollup": to false and keep the rollup=true in granularitySpec? would it still do the job for me and rollup during ingestion?Yes, it is ok to set forceGuaranteedRollup to false, and rollup to true. What this means is that data is rolled up within an individual segment, but not across segments
Suneet Saldanha
06/21/2023, 6:18 PMIf you want the data to be perfectly rolled up, you can change the partitioning spec to hashed or range (I think it's called single_dim on older versions of Druid), this will tell the ingestion spec to shuffle the data so that they are clustered more efficiently than in dynamic partitioning
Suneet Saldanha
06/21/2023, 6:19 PMwe are feeding our datasource via kafka supervisor and it is set to fully compacted. after I run index_parallel, it would then do the job of ""forceGuaranteedRollup":true" yeah?I don't understand this. Are you using the index_parallel job to compact the data, or do you have auto-compaction scheduled for the datasource and this index_parallel job is just some backfill?