This message was deleted.

manual compaction?

Manual compaction

please paste the compaction spec here

{ "type": "kafka", "spec": { "dataSchema": { "dataSource": "responseLog", "timestampSpec": { "column": "timestamp", "format": "auto", "missingValue": null }, "dimensionsSpec": { "dimensions": [ { "type": "string", "name": "domain", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "string", "name": "device", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "string", "name": "adUnitId", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "string", "name": "supplyPartner", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "string", "name": "supplyPartnerId", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "string", "name": "demandPartner", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "string", "name": "demandPartnerId", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "string", "name": "tenantId", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true } ], "dimensionExclusions": [ "__time", "requestId", "requestIdIn", "dspTxnId", "TotalAdResponse", "timestamp" ], "includeAllDimensions": false }, "metricsSpec": [ { "type": "thetaSketch", "name": "TotalAdResponse", "fieldName": "dspTxnId", "size": 16384, "shouldFinalize": true, "isInputThetaSketch": false, "errorBoundsStdDev": null }, { "type": "thetaSketch", "name": "requestIdIn", "fieldName": "requestId", "size": 16384, "shouldFinalize": true, "isInputThetaSketch": false, "errorBoundsStdDev": null } ], "granularitySpec": { "type": "uniform", "segmentGranularity": "DAY", "queryGranularity": "DAY", "rollup": true, "intervals": [] }, "transformSpec": { "filter": null, "transforms": [] } }, "ioConfig": { "topic": "responseLog", "inputFormat": { "type": "json", "flattenSpec": null, "featureSpec": {}, "keepNullColumns": false }, "replicas": 1, "taskCount": 1, "taskDuration": "PT1200S", "consumerProperties": { "bootstrap.servers": "192.168.1.1899092,192.168.1.1069092,192.168.1.185:9092" }, "autoScalerConfig": null, "pollTimeout": 100, "startDelay": "PT5S", "period": "PT30S", "useEarliestOffset": true, "completionTimeout": "PT1240S", "lateMessageRejectionPeriod": null, "earlyMessageRejectionPeriod": null, "lateMessageRejectionStartDateTime": null, "stream": "responseLog", "useEarliestSequenceNumber": true, "type": "kafka" }, "tuningConfig": { "type": "kafka", "appendableIndexSpec": { "type": "onheap", "preserveExistingMetrics": false }, "maxRowsInMemory": 1000000, "maxBytesInMemory": 0, "skipBytesInMemoryOverheadCheck": false, "maxRowsPerSegment": 5000000, "maxTotalRows": null, "intermediatePersistPeriod": "PT5M", "basePersistDirectory": "/home/ec2-user/bharat/apache-druid-24.0.0/var/tmp/druid-realtime-persist9903246862918681507", "maxPendingPersists": 0, "indexSpec": { "bitmap": { "type": "roaring", "compressRunOnSerialization": true }, "dimensionCompression": "lz4", "metricCompression": "lz4", "longEncoding": "longs", "segmentLoader": null }, "indexSpecForIntermediatePersists": { "bitmap": { "type": "roaring", "compressRunOnSerialization": true }, "dimensionCompression": "lz4", "metricCompression": "lz4", "longEncoding": "longs", "segmentLoader": null }, "reportParseExceptions": false, "handoffConditionTimeout": 0, "resetOffsetAutomatically": false, "segmentWriteOutMediumFactory": null, "workerThreads": null, "chatThreads": null, "chatRetries": 8, "httpTimeout": "PT10S", "shutdownTimeout": "PT80S", "offsetFetchPeriod": "PT30S", "intermediateHandoffPeriod": "P2147483647D", "logParseExceptions": false, "maxParseExceptions": 2147483647, "maxSavedParseExceptions": 0, "skipSequenceNumberAvailabilityCheck": false, "repartitionTransitionDuration": "PT120S" } }, "context": null }

This is the kafka ingestion spec. You had issue with compaction?

Yes

please get me the spec for the compaction task

Given

I thought you were using manual compaction? Paste the full json here

Will paste in 10 mins

instead of directly editing json just specify the attributes in the form

I do here only

So the error is when you enable auto compaction?

I am still not Getting . I am not seeing option of auto compaction . I just edit rule and submit

That is auto compaction. So the error is when you submit?

No once i submit it shows running and after some time I see failed compact task in task list

Paste the task log here

I am unable to see task log it shows. 404 error

What is your deep storage?

Local

Is this a single node or multi node setup

Mukti mode

With multi node setup you need either s3 or a network file system as deep storage. Local will not work. All nodes must be able to access all the segments in deep storage.