Hi Team, we have datasource collecting 10TB data per day with close to 80k segments with daily segment granularity (day). Just would like to know, do we need 10 TB of diskspace space to compact them or keeping lower disks volumes would be sufficient as 128GB ?

The middlemanagers / indexers - it depends on how many are being used by the compaction tasks. For historicals, it is slightly complicated. In druid, versions are immutable and atomically replaced If the compaction produces 8TB worth of data, you'd need that much free space across all historicals, so that they are loaded, and then the old data is dropped successfully

Deep Storage would also need this space

Thanks. I’m clear on historicals and deep storage part. Little confused on middlemenagers, do you mean compaction load gets splitted equally depending on number of compaction middlemanagers that I have to compact this data ? Suppose if I have 10 middlemanagers, each middle manager should have 1tb to process whole 10 TB ? Now that we only have segment granularity as day, how will compaction splits the segments ? I observed, previously we have hourly segments with 80k now that got reduced 15k after changing to segment granularity to day and query to none. Will it cause any query response improvement or decrement ?

The atomic unit is the segment file. If there's enough data to be processed then it will be split across all of your middle-managers. It won't necessarily be split evenly across hosts but it should be split evenly across available slots. You may end up using much less than 10 TB depending on how the work gets split. See maxNumConcurrentSubTasks in

tuningConfig

See also splitHintSpec and partitionsSpec in

tuningConfig

If you have 10 hosts and plenty of available slots, I would use a config that looks something like this:

"tuningConfig" : {
    "type": "index_parallel",
    "maxNumConcurrentSubTasks": 10,  // Max number of concurrent tasks. This must be > 1 to enable parallel indexing. Be careful not to over-allocate your available slots or you may block ingestion tasks.
    "splitHintSpec": {
      "type": "maxSize",
      "maxSplitSize": 1073741824,    // Default is 1GiB
      "maxNumFiles": 1000            // Default is 1000
    },
    "partitionsSpec": {
      "type": "dynamic",             // Fastest strategy but has imperfect rollup.
      "maxRowsPerSegment": 5000000,  // Target segment sizes < 1 GiB. Default is 5000000
    }
  },

Phase 1: With 10 TiB to read; the first phase will create ~10,240 tasks and start by assigning the first 10 (up to

maxNumConcurrentSubTasks

) tasks to run in parallel and assign additional tasks to worker threads as they free up. Phase 2: An estimate of how many rows will be created is established in phase 1. Assuming we end up with ~75 billion rows; the second phase will create ~15,000 tasks and will work as assigned similar to phase 1. Since you have 15K segment files per day; I would strongly recommend a smaller segment granularity such as

hour

or even

five_minute

. Having ~50 segment files per 5 minutes will still have a nice split across your historical servers. I'm unsure how changing your segment granularity from

hour

to

day

results in fewer segments, unless you weren't compacting your data until now or are using rollup with

queryGranularity=day

. Regarding performance; typically fewer files means less disk I/O and therefore better performance. Depending on your use case, if you can rollup the data this could result in significant space and query time savings. Be aware this truncates your timestamp (

__time

field) and comes with other considerations on how to roll up your metrics. For ideal rollup, I would set

segmentGranularity

>=

queryGranularity

Good luck and happy tuning!