This message was deleted.

As you have 24 segments that suggest, you might have segment granularity as Hour, if you compact it using Day granularity, then you should see 1 segment only

okay great thanks, can we do the compaction configuration using the UI instead of spec.?

Auto compaction, you can do from console but you will not be able to change all parameters, ssystem will do the best compaction available, If you want to manually change something like here you want 24 segments to be 1 then you should do manual compaction through SPEC.

Okay thanks a lot.!

The interval will tell the compaction task to compact all segments within the interval. Complete details on compaction with example is given in below link : https://druid.apache.org/docs/latest/data-management/compaction.html

ok thanks

You can submit it as a normal task from the ingestion section, the datasource is their in SPEC.

ok

No, compaction does not do data loss.

okay,thats a relief, thanks, but what about the reduced size and no of rows, my no of rows reduced to almost half of that before compaction

That is expected if you have rollup enabled or query granularity changed. Below links has the details on the roll up and query granularity, check both ingestion and compaction spec. or you can query to the table now and see if there is any data loss. https://druid.apache.org/docs/latest/data-management/compaction.html#setting-up-manual-compaction https://druid.apache.org/docs/latest/data-management/compaction.html Snippet from attached link:

By default, compaction does not modify the underlying data of the segments. However, there are cases when you may want to modify data during compaction to improve query performance:

If, after ingestion, you realize that data for the time interval is sparse, you can use compaction to increase the segment granularity.
If you don't need fine-grained granularity for older data, you can use compaction to change older segments to a coarser query granularity. For example, from minute to hour or hour to day. This reduces the storage space required for older data.
You can change the dimension order to improve sorting and reduce segment size.
You can remove unused columns in compaction or implement an aggregation metric for older data.
You can change segment rollup from dynamic partitioning with best-effort rollup to hash or range partitioning with perfect rollup. For more information on rollup, see perfect vs best-effort rollup.

"type": "compact",
  "dataSource": "test",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "Provide your interval"
    }
  },
  "granularitySpec": {
    "segmentGranularity": "DAY",
  }
}

i used this spec which you gave and changed the segment granularity to month cause it is set to month in my original table the task duration for my original table is 1H

In this spec it is not present, it would be a separate parameter.

yes initial ingestion does have a rollup

when you run the task beside to the task there will be option to see the logs, let me provide a screen shot

okay, will check , thanks a lot.!!

If it shows fully compacted that means the compaction has run, I have seen in some situation the first ever compaction does not show iin the task log, if you want you can run again and see the task log. Just test with any small datasource.

compaction has run, but it is not making the no of segments i wish to make, i need a segment size of around 400mb I am getting around 1 -2 mb

This totally depends on your data, you are using segmentGranularity as day in compaction, and if you have less data for each day, then compacting with day granularity will not help, may be try with month granularity. This totally depends on data and there isn’t only one setting, we need to understand the parameters that is present for the compaction and accordingly configure manual compaction.

I had set it to month for tables with less data

I think during ingestion you can set based on size but if you have less data per day and you are using day segment granularity then one segment will get created for that day, if you have more data only size will come into picture.