Hi Everyone, we are observing segments are created...
# generals
sriramdas sivasai
09/29/2024, 12:46 AMHi Everyone, we are observing segments are created hourly although the segment granularity that i set is day. attaching the tuning config below, could some help me on this ?
sriramdas sivasai
09/29/2024, 12:58 AM{
"type": "kafka",
"spec": {
"dataSchema": {
**
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "DAY",
"queryGranularity": {
"type": "none"
},
"rollup": false,
"intervals": []
},
"transformSpec": {
"filter": null,
"transforms": []
}
},
"ioConfig": {
"topic": "**",
"topicPattern": null,
"inputFormat": {
"type": "json",
"keepNullColumns": false,
"assumeNewlineDelimited": false,
"useJsonNodeReader": false
},
"replicas": 1,
"taskCount": 48,
"taskDuration": "PT7200S",
"consumerProperties": {
"bootstrap.servers": "**",
"group.id": "dmdaf-druid"
},
"autoScalerConfig": null,
"pollTimeout": 100,
"startDelay": "PT5S",
"period": "PT30S",
"useEarliestOffset": false,
"completionTimeout": "PT14400S",
"lateMessageRejectionPeriod": "PT86400S",
"earlyMessageRejectionPeriod": null,
"lateMessageRejectionStartDateTime": null,
"configOverrides": null,
"idleConfig": null,
"stopTaskCount": 48,
"stream": "dmdaf-**",
"useEarliestSequenceNumber": false,
"type": "kafka"
},
"tuningConfig": {
"type": "kafka",
"appendableIndexSpec": {
"type": "onheap",
"preserveExistingMetrics": false
},
"maxRowsInMemory": 1500000,
"maxBytesInMemory": 0,
"skipBytesInMemoryOverheadCheck": false,
"maxRowsPerSegment": 1500000,
"maxTotalRows": null,
"intermediatePersistPeriod": "PT120M",
"maxPendingPersists": 0,
"indexSpec": {
"bitmap": {
"type": "roaring"
},
"dimensionCompression": "lz4",
"stringDictionaryEncoding": {
"type": "utf8"
},
"metricCompression": "lz4",
"longEncoding": "longs"
},
"indexSpecForIntermediatePersists": {
"bitmap": {
"type": "roaring"
},
"dimensionCompression": "lz4",
"stringDictionaryEncoding": {
"type": "utf8"
},
"metricCompression": "lz4",
"longEncoding": "longs"
},
"reportParseExceptions": false,
"handoffConditionTimeout": 0,
"resetOffsetAutomatically": false,
"segmentWriteOutMediumFactory": null,
"workerThreads": null,
"chatRetries": 20,
"httpTimeout": "PT30S",
"shutdownTimeout": "PT80S",
"offsetFetchPeriod": "PT30S",
"intermediateHandoffPeriod": "P2147483647D",
"logParseExceptions": false,
"maxParseExceptions": 2147483647,
"maxSavedParseExceptions": 0,
"skipSequenceNumberAvailabilityCheck": false,
"repartitionTransitionDuration": "PT120S"
}
},
"context": null
}
we recieve Billion of the events and lot of them are of previous hours (may be in last 1 days) we see high cpu usage per each task
a
Amatya Avadhanula
09/29/2024, 6:02 AMHi Sivasai.
Appending tasks use the configured segment granularity for the first segment of the interval
Amatya Avadhanula
09/29/2024, 6:02 AMIf an hourly segment already exists, Kafka tasks will use the existing granularity and append on top of the segments that already exist
s
sriramdas sivasai
09/29/2024, 6:04 AMI'm using druid 29.0.0. we observe changes to segment granularity for the druid data sources is not applying even after we did suspend and restart + even after reset of the supervisor.
initially it was hour and later i changed to day
but creating the new datasource with same config is creating with day level granularity
sriramdas sivasai
09/29/2024, 6:05 AMis there any way to change this ?Copy codeIf an hourly segment already exists, Kafka tasks will use the existing granularity and append on top of the segments that already exist
a
Amatya Avadhanula
09/29/2024, 6:05 AMYes. It doesn't have much to do with the supervisor spec, but with the existing segment granularity
Amatya Avadhanula
09/29/2024, 6:05 AMIf you want to change it, pause Kafka ingestion and reindex the existing datasource to day granularity
Amatya Avadhanula
09/29/2024, 6:05 AMYou can then submit the supervisor again with day granularity
s
sriramdas sivasai
09/29/2024, 6:06 AMdo i have to do it for all the dates or just the latest date would work ?
sriramdas sivasai
09/29/2024, 6:06 AMor shall i wait for day or 2, will it creates day level segments in the new dates ?
a
Amatya Avadhanula
09/29/2024, 6:07 AMDo you want day level granularity across your entire datasource or just for new intervals?
Amatya Avadhanula
09/29/2024, 6:08 AMAlso, one way to avoid needing to reindex the data manually is to set up automatic compaction for the datasource with Day granularity
s
sriramdas sivasai
09/29/2024, 6:08 AMim okay with the old data that was created. but new day i would like to start writing it as day.
a
Amatya Avadhanula
09/29/2024, 6:09 AMThat will happen automatically for the new data intervals for which no segments exist
s
sriramdas sivasai
09/29/2024, 6:11 AMthat's cool. just to confirm,
suppose if we dont have any segments for the day sep 30 so far and new segment that creates for sept 30 going to have day wise segments and may be if we see some sept 29 data then it can create hourly segment (WITH OUT any reindexing / compaction ) .
Could you please correct me if im wrong ?
a
Amatya Avadhanula
09/29/2024, 6:12 AMThat understanding is correct.
s
sriramdas sivasai
09/29/2024, 6:12 AMGreat. Thanks for your response !!
👍 1
2 Views