I have a question about how to speed up the compaction process for a datasource with hash-based partitions. I have a datasource called SubscriberEvents with a segment granularity of 12 hours. This datasource also uses hashed partitions with numShards set to 100, and the partition dimension is something like SUBSCRIBER_ID. I want to double the segment granularity to 1 day and increase the numShards from 100 to 200. The goal is to maintain the existing segment size while improving the scan performance for queries based on SUBSCRIBER_ID, since scanning 1 day of data will only involve one segment instead of two. However, the existing datasource is huge, with 12 hours of data containing about 1 billion events (2 billion per day). I expect the compaction process to take a very long time, considering that ingesting 12 hours of events into Druid typically takes around 8 hours. My question is: How will Druid handle this compaction? Will it need to scan all 100*2 segments to calculate 200 partitions and then build 1-day granularity segments with 200 partitions? Or is there a possibility that Druid will perform the compaction on a per-segment basis? For example, could it split the first 12-hour segment (from 00:00 to 11:59) into two parts based on SUBSCRIBER_ID, and then concatenate each of these with the corresponding split parts of the subsequent segment (from 12:00 to 23:59)?

To speed up the compaction process for your datasource

SubscriberEvents

, which currently has a segment granularity of 12 hours and uses hash-based partitions, it's important to understand how Druid handles compaction tasks, especially when changing parameters like segment granularity and number of shards. When you initiate a compaction process in Druid, it typically operates on an entire time interval covered by segments. In your case, if you have existing segments for the 12-hour period containing 1 billion events, the compaction will involve reading through these segments to create the new segments with 1-day granularity and the adjusted number of shards. 1. Segment Scanning Requirement: Yes, Druid will indeed need to scan all existing segments, which means it will look at the 100 segments corresponding to the 12-hour granularity. Each of these segments will be referenced as Druid calculates the new partitions for the 1-day granularity. The result will be 200 segments for each day, effectively doubling your number of shards. 2. Compaction Process: Druid performs compaction in a way that it processes segments at once, rather than on a per-segment basis when transitioning from a smaller segment size to a larger one. This means it will read data from all 100 corresponding segments of the previous granularity in order to compute the new segments. This is necessary to ensure that the partitioning by

SUBSCRIBER_ID

is correctly maintained across the entire data set. 3. Efficiency Considerations: While the process can be time-consuming due to the sheer volume of data (1 billion events for 12 hours), Druid is optimized for such tasks. The compaction will focus on reorganizing your data effectively, but because you're dealing with a large amount of data and a change in partitioning, you might need to plan the execution during off-peak hours to mitigate performance degradation during the compaction. 4. Possibility of Per-Segment Handling: Druid is unlikely to split and concatenate on a per-segment basis as you suggested. Instead, it will aggregate data at a broader level throughout all the relevant segments in the specified time range. This ensures that all data contributing to a partition is considered in the final compacted segment, maintaining integrity and performance. Thus, while the compaction process will indeed take a significant amount of time given the data volume, Druid optimizes the data restructuring to enhance query performance effectively after compaction is completed. You can expect improved scanning performance for queries based on

SUBSCRIBER_ID

post-compaction. Sources: - Compaction Overview: compaction.md (internal document) - Automatic Compaction Details: automatic-compaction.md (internal document) - Managing Partitioning Schemes: multitenancy.md (internal document)