https://linen.dev logo
Join Slack
Powered by
Hi Team, we have been facing an issue with the com...
# general
s
Hi Team, we have been facing an issue with the compaction with one of our streams throwing 
No Space left of the device
where we have configured segment granularity as 1 hour, volume of the data for 1 hour is of 300GB and disk attached to our middle manager is only of 40GB. all though we set the max_total_rows (10M records), volume of 10GB, its still failing with 
No Space left of the device
. One option that I see here is to increase the disk size to 300GB ? Is there any config that i can apply to pull only some records, compact and handoff with disk space of 40GB ?
j
Hi Sivasai, The only thing I can think of is to use MSQ to copy the data over to a new temporary datasource with finer segmentGranularity, and do it in smaller chunks. For example: REPLACE INTO tempTable overwrite where [some 5 minute interval] select * from originalTable where [__time in that 5 minute interval] partitioned by TIME_FLOOR(__time, 'PT5M’)
You could probably do this with native reindexing spec as well.
s
Okay thanks for the suggestions. Wouldnt we have any possible way to tweak/set the limitations for input of the compaction tasks ? Previously i could think of is to use 
inputSegmentSizeBytes
- but right now its deprecated (any alternative for this ) ?. i was thinking of trying to use 
Split Hint Spec
to restrict the size of the input. would this work ?
a
At which phase do you see the failure?
i.e during the first phase for cardinality or later during generate or merge?
s
from : appenderator-merge: attaching log snippet
Copy code
2024-05-20T04:24:40,021 WARN [task-runner-0-priority-0] org.apache.druid.indexing.input.DruidSegmentInputEntity - Could not clean temporary segment file: /opt/druid/var/druid/task/slot0/coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z/work/indexing-tmp/beta_ds/2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z/2024-05-17T18:56:40.935Z/203
2024-05-20T04:24:40,024 INFO [task-runner-0-priority-0] org.apache.druid.storage.s3.S3DataSegmentPuller - Pulling index at path[CloudObjectLocation{bucket='bcm_druid_vault_prod_1', path='druid/segments/beta_ds/2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z/2024-05-17T18:56:40.935Z/204/cc5aa6bf-e264-4c5f-8e46-297c00648682/index.zip'}] to outDir[/opt/druid/var/druid/task/slot0/coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z/work/indexing-tmp/beta_ds/2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z/2024-05-17T18:56:40.935Z/204]
2024-05-20T04:24:42,755 INFO [task-runner-0-priority-0] org.apache.druid.storage.s3.S3DataSegmentPuller - Loaded 376548982 bytes from [CloudObjectLocation{bucket='bcm_druid_vault_prod_1', path='druid/segments/beta_ds/2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z/2024-05-17T18:56:40.935Z/204/cc5aa6bf-e264-4c5f-8e46-297c00648682/index.zip'}] to [/opt/druid/var/druid/task/slot0/coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z/work/indexing-tmp/beta_ds/2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z/2024-05-17T18:56:40.935Z/204]
2024-05-20T04:24:44,941 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushing in-memory data to disk because (estimated) bytesCurrentlyInMemory[1431658712] is greater than maxBytesInMemory[1431655765].
2024-05-20T04:25:01,777 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37] spill[1] to disk in [43,274] ms (434,406 rows).
2024-05-20T04:25:01,797 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data with commit metadata [null] for segments: beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37
2024-05-20T04:25:01,797 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted stats: processed rows: [56803035], persisted rows[434406], sinks: [1], total fireHydrants (across sinks): [2], persisted fireHydrants (across sinks): [2]
2024-05-20T04:25:01,797 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Segment[beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37] hydrant[FireHydrant{queryable=null, count=1}] already swapped. Ignoring request to persist.
2024-05-20T04:25:01,798 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Ingestion was throttled for [16,854] millis because persists were pending.
2024-05-20T04:25:01,798 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted rows[434,011] and (estimated) bytes[1,431,354,356]
2024-05-20T04:25:11,244 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing [1] segments in background
2024-05-20T04:25:11,244 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing segments: [beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37]
2024-05-20T04:25:43,920 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37] spill[2] to disk in [42,121] ms (434,011 rows).
2024-05-20T04:25:43,937 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data with commit metadata [null] for segments: beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37
2024-05-20T04:25:43,937 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted stats: processed rows: [57000000], persisted rows[434011], sinks: [1], total fireHydrants (across sinks): [3], persisted fireHydrants (across sinks): [2]
2024-05-20T04:25:43,937 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Segment[beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37] hydrant[FireHydrant{queryable=null, count=2}] already swapped. Ignoring request to persist.
2024-05-20T04:25:43,937 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Ingestion was throttled for [32,693] millis because persists were pending.
2024-05-20T04:25:43,937 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted rows[196,965] and (estimated) bytes[686,828,106]
2024-05-20T04:26:03,484 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37] spill[3] to disk in [19,546] ms (196,965 rows).
2024-05-20T04:26:03,492 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data with commit metadata [null] for segments: beta_ds_2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z_2024-05-20T00:20:08.411Z_37
2024-05-20T04:26:03,492 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted stats: processed rows: [57000000], persisted rows[196965], sinks: [1], total fireHydrants (across sinks): [4], persisted fireHydrants (across sinks): [2]
2024-05-20T04:26:03,492 INFO [[coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Preparing to push (stats): processed rows: [57000000], sinks: [1], fireHydrants (across sinks): [4]
2024-05-20T04:27:40,649 INFO [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution
java.io.IOException: No space left on device
at java.io.FileDescriptor.close0(Native Method) ~[?:?]
at java.io.FileDescriptor.close(FileDescriptor.java:297) ~[?:?]
at java.io.FileOutputStream$1.close(FileOutputStream.java:392) ~[?:?]
at java.io.FileDescriptor.closeAll(FileDescriptor.java:355) ~[?:?]
at java.io.FileOutputStream.close(FileOutputStream.java:390) ~[?:?]
at org.apache.druid.indexing.overlord.ForkingTaskRunner.waitForTaskProcessToComplete(ForkingTaskRunner.java:513) ~[druid-indexing-service-29.0.0.jar:29.0.0]
at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:404) ~[druid-indexing-service-29.0.0.jar:29.0.0]
at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:171) ~[druid-indexing-service-29.0.0.jar:29.0.0]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-32.0.1-jre.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-05-20T04:27:47,626 INFO [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: /opt/druid/var/druid/task/slot0/coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z
I'm extremely seeing this warning, Suspecting its not clearing unrequired data on the disk
Copy code
2024-05-20T00:59:51,246 WARN [task-runner-0-priority-0] org.apache.druid.indexing.input.DruidSegmentInputEntity - 
Could not clean temporary segment file: /opt/druid/var/druid/task/slot0/coordinator-issued_compact_beta_ds_jgfdclmh_2024-05-20T00:20:07.961Z/work/indexing-tmp/beta_ds/2024-05-17T18:00:00.000Z_2024-05-17T19:00:00.000Z/2024-05-17T18:56:40.935Z/2
Here is the compaction spec that i use
Copy code
{
  "dataSource": "beta_ds",
  "taskPriority": 25,
  "inputSegmentSizeBytes": 100000000000000,
  "maxRowsPerSegment": null,
  "skipOffsetFromLatest": "P1D",
  "tuningConfig": {
    "maxRowsInMemory": null,
    "appendableIndexSpec": null,
    "maxBytesInMemory": null,
    "maxTotalRows": null,
    "splitHintSpec": {
      "type": "maxSize",
      "maxSplitSize": 5368709120,
      "maxNumFiles": 100
    },
    "partitionsSpec": {
      "type": "dynamic",
      "maxRowsPerSegment": 1500000,
      "maxTotalRows": 5000000
    },
    "indexSpec": null,
    "indexSpecForIntermediatePersists": null,
    "maxPendingPersists": null,
    "pushTimeout": null,
    "segmentWriteOutMediumFactory": null,
    "maxNumConcurrentSubTasks": null,
    "maxRetry": null,
    "taskStatusCheckPeriodMs": null,
    "chatHandlerTimeout": null,
    "chatHandlerNumRetries": null,
    "maxNumSegmentsToMerge": null,
    "totalNumMergeTasks": null,
    "maxColumnsToMerge": null,
    "type": "index_parallel",
    "forceGuaranteedRollup": false
  },
  "granularitySpec": null,
  "dimensionsSpec": null,
  "metricsSpec": null,
  "transformSpec": null,
  "ioConfig": null,
  "taskContext": null
}
@Amatya Avadhanula any pointers on this ?
a
For some reason I misread it as compaction failing with OOD
Is your kafka task failing with OOD?
s
No. compaction failing with OOD. not the kafka task !!!
a
What is the average row size?
s
888 bytes
a
Could you try reducing the following?: splitHintSpec.maxSplitSize to 2 GB partitionsSpec.maxRowsPerSegment to 250000 partitionsSpec.maxTotalRows to 500000
s
tried making this change. No luck !!!. is there any alternative to inputSegmentSizeBytes ?
Open in Slack
PreviousNext
Powered by