This message was deleted.
# troubleshootings
Slackbot
05/23/2023, 1:50 PMThis message was deleted.
s
Suneet Saldanha
05/23/2023, 2:40 PMThe error appears to state that Druid doesn't know how to resolve 2 conflicting aggregators for the column
mt_l_uppts_inferece_avgSuneet Saldanha
05/23/2023, 2:41 PMYou can resolve this by explicitly telling Druid what the metricsSpec should be in the auto-compaction spec https://druid.apache.org/docs/latest/data-management/automatic-compaction.html#configure-automatic-compaction
Suneet Saldanha
05/23/2023, 2:43 PMin fact, if you know the granularitySpec, dimensionsSpec and metricsSpec of the datasource - you can specify this in the auto compaction config and it should speed up your compaction jobs, as they no longer need to download segments to infer the schema.
Copy code
Compaction tasks fetch all relevant segments prior to launching any subtasks, unless the following items are all set. It is strongly recommended to set all of these items to maximize performance and minimize disk usage of the compact tasks launched by auto-compaction:o
Olivier JOUSSET
05/23/2023, 3:05 PMOk so we have to specify manually the metricSpec when configuring the AutoCompaction ? Is there any conf to indicate to use the latest segment schema as reference for instance ?
Olivier JOUSSET
05/23/2023, 3:10 PManother question do you know if this conflict is only for a metric
types
Suneet Saldanha
05/23/2023, 3:11 PMIs there any conf to indicate to use the latest segment schema as reference for instance ?I'm not aware of one, but that would be nice!
Suneet Saldanha
05/23/2023, 3:14 PMOk so we have to specify manually the metricSpec when configuring the AutoCompaction ?To resolve this issue via auto-compaction, yes. But you can also issue a manual compact task / a Druid re-index task to fix the time interval that auto-compaction is complaining about, and leave the auto-compaction config unchanged.
Suneet Saldanha
05/23/2023, 3:21 PMn do you know if this conflict is only for a metricAdding or removing metrics should work seamlessly. This exception is only thrown when 2 aggregator types are conflicting for the same column, so Druid is unsure how to resolve the difference. In this instance I think there is the chance. https://github.com/apache/druid/blob/338bdb35ea19b495bc6eb587ab99978edb16ce46/proc[…]/java/org/apache/druid/query/aggregation/AggregatorFactory.javaupdate are for any schema update (add/remove metrics/dimension) ?type
Suneet Saldanha
05/23/2023, 3:23 PMI'm unsure how conflicting dimension types are resolved in compaction
https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java#L1078 - this method seems to be the one that handles the dimensions spec from different segments.
Suneet Saldanha
05/23/2023, 3:23 PMHope this helps
o
Olivier JOUSSET
05/23/2023, 3:50 PMok thanks
d
Didip Kerabat
05/23/2023, 8:50 PMTo avoid issues like this, we usually start compaction on much older dates, e.g. 2 weeks older from today because our customers usually already update incorrect schema and performed backfills within those 2 weeks.
o
Olivier JOUSSET
05/24/2023, 8:59 AMOk but actually whatever the exclusion period you'll always have this conflicts if 2 segments with different schema have to be conpacted no ?
d
Didip Kerabat
05/24/2023, 9:52 PMyes, correct. Not bulletproof. This will happen anytime you have 2 different schema in the same compaction time range.
✅ 1