Hi druids! My team is coming across a data integrity issue using Druid. We've got Kafka as input and the same topics serving Druid and Snowflake. In Snowflake we get full granularity of the data and in Druid we get it rolled up. The problem is that Druid is creating duplicates and rolling them up, leading to metrics doubling their expected values. This happens to a small percentage of the total ingested data but it's having a negative effect. Did anyone else come across this issue before? How can this be mitigated? Is this a "best-effort rollup" limitation? Essentially the column publisher revenue gbp in Druid should be 0.0045 instead of 0.009 based on the rows ingested. You can see that the column count shows the number of rolled-up rows. Thanks

I've not seen this before. Do you have the means to double-check the source data (i.e. Kafka) to double-check that it isn't Snowflake that is dropping data?