This message was deleted.

Yes. Compaction will create a new set of segment files that replace existing segments. While it builds the new set, the old set are available for query.

But just to be clear: compaction happens in the background and is totally "invisible" from the query side (other than it usually improves performance). Very notably it does not change the results of any query (if it does it would be a big issue). You need to make sure that your data model makes sense without thinking about compaction

you considered data to be duplicate if a single row is equal in all values to another row.

it is precisely this

ok, what about 1 and 2?

those are true as well

One question: When the auto-compaction is enabled for a datasource, will the original segments (without compaction) be kept intact forever? From the above discussion I assume, no. If so, when will the old segments be discarded?

if you mean for the original segment, in the object storage, yes, it's kept until the 'kill task' is issued

it's kept until the 'kill task' is issued

Yes, that seems to be true. But the dilemma that I've with Kill Task is - how to differentiate between 1. the segments marked as unused after compaction 2. those segments that are marked as unused(dropped) by a Data Retention Rule ? I do not want to delete the segments marked as unused because of Data Retention Rule.

yeah, you kinda need to run the kill before the retention rule go in

Renato, could you expand on this? Did the kill job fail to remove the unused segments from S3?

Hello, @Sergio Ferragut, I thought that the kill task was only necessary to remove older data after the retention period, as I don't want to remove any data, I never ran any kill task, but I was sure that the compaction jobs would remove the older versions automatically (as well failed tasks that leave unfinished segments in the s3, eg: when a mm fails with OOM) I think Clinton had the same impression that this cleanup maintenance would be included by default, and not affect unused segments not loaded because of the retention period

I see. So perhaps an improvement would be to separate kill for cleanup vs kill for “dropped” segments data on retention.

@Sergio Ferragut: The assumption that I had when I ran compaction task was that, the older smaller segments will be removed. But Druid keeps both segments. with Kill Task - how to differentiate between 1. the segments marked as unused after compaction 2. those segments that are marked as unused(dropped) by a Data Retention Rule ? Do not want to delete the segments marked as unused because of Data Retention Rule.

for who was not aware of what "30W" means, according to chatGPT, 30W means:

The term "30W" is a notation used in China to represent 300,000 units. The W stands for "wan", which is used in the Chinese language to denote ten thousand. Therefore, "30W" stands for 30 * 10,000, which equals 300,000. So, in your context, there are more than 300,000 segment records in metadata storage.

Thanks @Hareesh Joshi for the issue and @Renato Santos for the 30W explanation (🙂 I was thinking it was an upside M). There is ongoing work in the Apache Druid project to figure out cold query capability, this is the idea of having used segments that are not cached in historicals and live in deep storage only. I think this will dovetail nicely to distinguish between cleanup and retention based segments, so that cleanup can occur regularly without removing data that you don't want to lose but also don't want to cache.