https://linen.dev logo
Join Slack
Powered by
I've been noticing in the druid re-index task logs...
# general
j
I've been noticing in the druid re-index task logs I get these warnings repeatedly:
Copy code
2024-09-27T14:25:41,788 WARN [task-runner-0-priority-0] org.apache.druid.indexing.input.DruidSegmentInputEntity - Could not clean temporary segment file: /data/druid/apache-druid-29.0.1/var/druid/task/slot2/index_parallel_datasource_madlbnge_2024-09-26T08:22:48.343Z/work/indexing-tmp/datasource/2023-07-31T00:00:00.000Z_2023-08-01T00:00:00.000Z/2023-08-11T17:03:31.432Z/17
When the task completes, the files are always cleared; but while the task is running it appears to be unable to delete these files. I double-checked the file permissions and the task has all the permissions necessary. In fact, its the job that created the files in the first place so I'm unsure why I get these warnings in the logs. This has been an issue for a while but since the files were always cleared after the job completed I didn't bother raising it. However, this is now impairing our cluster as disk space is running out and large jobs are failing. I feel like if this issue could be resolved then the reindex tasks could clean the disk as it goes rather than running out of disk space.
Here's another example:
Copy code
2024-09-27T15:12:06,168 INFO [task-runner-0-priority-0] org.apache.druid.segment.loading.LocalDataSegmentPuller - Copied 70906675 bytes from [/data/druid/deep-storage/segments/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/18/index] to [/data/druid/apache-druid-29.0.1/var/druid/task/slot0/partial_dimension_cardinality_datasource-volumes_cagnjbhb_2024-09-27T15:09:30.478Z/work/indexing-tmp/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/18]
2024-09-27T15:12:19,399 WARN [task-runner-0-priority-0] org.apache.druid.indexing.input.DruidSegmentInputEntity - Could not clean temporary segment file: /data/druid/apache-druid-29.0.1/var/druid/task/slot0/partial_dimension_cardinality_datasource-volumes_cagnjbhb_2024-09-27T15:09:30.478Z/work/indexing-tmp/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/18
2024-09-27T15:12:19,923 INFO [task-runner-0-priority-0] org.apache.druid.segment.loading.LocalDataSegmentPuller - Copied 71312687 bytes from [/data/druid/deep-storage/segments/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/19/index] to [/data/druid/apache-druid-29.0.1/var/druid/task/slot0/partial_dimension_cardinality_datasource-volumes_cagnjbhb_2024-09-27T15:09:30.478Z/work/indexing-tmp/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/19]
2024-09-27T15:12:32,868 WARN [task-runner-0-priority-0] org.apache.druid.indexing.input.DruidSegmentInputEntity - Could not clean temporary segment file: /data/druid/apache-druid-29.0.1/var/druid/task/slot0/partial_dimension_cardinality_datasource-volumes_cagnjbhb_2024-09-27T15:09:30.478Z/work/indexing-tmp/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/19
2024-09-27T15:12:33,241 INFO [task-runner-0-priority-0] org.apache.druid.segment.loading.LocalDataSegmentPuller - Copied 55235500 bytes from [/data/druid/deep-storage/segments/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/20/index] to [/data/druid/apache-druid-29.0.1/var/druid/task/slot0/partial_dimension_cardinality_datasource-volumes_cagnjbhb_2024-09-27T15:09:30.478Z/work/indexing-tmp/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/20]
2024-09-27T15:12:42,927 WARN [task-runner-0-priority-0] org.apache.druid.indexing.input.DruidSegmentInputEntity - Could not clean temporary segment file: /data/druid/apache-druid-29.0.1/var/druid/task/slot0/partial_dimension_cardinality_datasource-volumes_cagnjbhb_2024-09-27T15:09:30.478Z/work/indexing-tmp/datasource/2024-01-08T00:00:00.000Z_2024-01-09T00:00:00.000Z/2024-01-09T00:10:49.700Z/20
And another example:
Copy code
2024-09-27T15:18:58,032 INFO [task-runner-0-priority-0] org.apache.druid.segment.loading.LocalDataSegmentPuller - Unzipped 40672413 bytes from [/data/druid/deep-storage/segments/datasource/2023-04-29T00:00:00.000Z_2023-04-30T00:00:00.000Z/2023-04-30T03:18:43.772Z/5/index.zip] to [/data/druid/apache-druid-29.0.1/var/druid/task/slot8/index_parallel_datasource-volumes_ieindbak_2024-09-27T01:05:19.225Z/work/indexing-tmp/datasource/2023-04-29T00:00:00.000Z_2023-04-30T00:00:00.000Z/2023-04-30T03:18:43.772Z/5]
2024-09-27T15:20:15,713 WARN [task-runner-0-priority-0] org.apache.druid.indexing.input.DruidSegmentInputEntity - Could not clean temporary segment file: /data/druid/apache-druid-29.0.1/var/druid/task/slot8/index_parallel_datasource-volumes_ieindbak_2024-09-27T01:05:19.225Z/work/indexing-tmp/datasource/2023-04-29T00:00:00.000Z_2023-04-30T00:00:00.000Z/2023-04-30T03:18:43.772Z/5
Open in Slack
PreviousNext
Powered by