Hi Everyone, after performing manual compaction I noticed that the size of the datasource is increasing significantly(almost double). I tried the same repartitioning with autocompaction and there the size was not increasing, Does anyone know what the reason could be for this ?

Hi tvthomas ... can you share your ingestion and compaction specs? There are a few different things that could cause segment size to increase, but easier to start by looking at what is there.

sure John, This is the Ingestion Spec { "type": "index_parallel", "spec": { "ioConfig": { "type": "index_parallel", "inputSource": { "type": "s3", "uris": [ "s3://s3-ingestion/processed/2024/08/05/Ingestion_TESTING.csv" ] }, "inputFormat": { "type": "csv", "findColumnsFromHeader": false, "columns": [ "ServicePointID", "chanSubtype", "sensorTS", "val", "intv", "readFlags", "veeFlags", "readStatus", "SourceSystem", "offset", "local_ts", "creation_ts", "ingestion_ts", "file_name" ], "skipHeaderRows": 1 }, "appendToExisting": true }, "tuningConfig": { "type": "index_parallel", "partitionsSpec": { "type": "dynamic", "maxRowsPerSegment": 10000000 }, "maxRowsInMemory": 10000000, "maxNumConcurrentSubTasks": 10 }, "dataSchema": { "dataSource": "compaction_test_trial", "timestampSpec": { "column": "sensorTS", "format": "iso" }, "transformSpec": { "transforms": [ { "type": "expression", "expression": "(\"ServicePointID\")", "name": "servicePointId" }, { "type": "expression", "expression": "(\"val\")", "name": "value" }, { "type": "expression", "expression": "(\"intv\")", "name": "readingInterval" }, { "type": "expression", "expression": "(\"chanSubtype\")", "name": "channelNumber" }, { "type": "expression", "expression": "string _to_array(readStatus,' ')", "name": "readStatus" } ] }, "dimensionsSpec": { "dimensions": [ { "name": "servicePointId", "type": "string" }, { "name": "channelNumber", "type": "int" }, { "name": "creation_ts", "type": "long" }, { "name": "local_ts", "type": "long" }, { "name": "offset", "type": "string" }, { "name": "readStatus", "type": "string", "multiValueHandling": "ARRAY" }, { "name": "SourceSystem", "type": "string" }, { "name": "readFlags", "type": "string" }, { "name": "veeFlags", "type": "int" }, { "name": "value", "type": "float" }, { "name": "readingInterval", "type": "int" }, { "name": "file_name", "type": "string" } ] }, "granularitySpec": { "queryGranularity": "none", "rollup": false, "segmentGranularity": "day" } } } } The one I used for Autocompaction was { "dataSource": "compaction_trial_test", "tuningConfig": { "partitionsSpec": { "type": "hashed", "maxRowsPerSegment": 47079295, "partitionDimensions": [ "servicePointId", "channelNumber", "local_ts" ] }, "maxNumConcurrentSubTasks": 5 }, "skipOffsetFromLatest": "PT0H" } And the one for Manual compaction is { "type": "compact", "dataSource": "compaction_trial_test", "ioConfig": { "type": "compact", "inputSpec": { "type": "interval", "interval": "2024-08-01/2024-08-11" } }, "granularitySpec": { "segmentGranularity": "DAY" }, "tuningConfig": { "type": "index_parallel", "maxRowsInMemory": 10000000, "maxNumConcurrentSubTasks": 15, "partitionsSpec": { "type": "hashed", "partitionDimensions":["servicePointId","channelNumber","local_ts"] }, "workerCategory": "batch", "targetRowsPerSegment":47079295, "forceGuaranteedRollup": true, "indexSpec": { "bitmap": { "type": "roaring" }, "dimensionCompression": "lz4", "metricCompression": "lz4", "longEncoding": "auto" } } } The partitioning type and columns are the same , The original size of the data source was 16Gb and after autocompaction it became 16.5 and the same data in another table when put through manual compaction became 32gb.

How many segments, and segment sizes, in each case? Possible forceGuaranteedRollup = true has force more, smaller segments, losing economies of scale and created more high-cardinality dictionarys with highly overlapping values across the smaller segments. If that doesn't explain it, then can you try range compaction instead of Hash? I haven't used Hash much, range is generally recommended over Hash

The problem is that Manual compaction doesn't support range partitioning and the next best option I had was hashing. The hashing repartitioning was reducing the segment count but is the avg row size has doubled now.

Can you find a completed auto-compaction job for this table and copy out the payload? When I run manual compaction tasks I usually pull and existing payload like this, remove the task/group IDs from the spec, change the time interval and then sumit the job for manual compaction. This ensures I am not changing anything that might affect the compaction parameteres. Curious to see what happens if you do this and re-compact the time interval where you saw the large increase.