https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
i
That is correct, they are marked as unused
t
Thank you Ian. I will dig to find out why they are not removed them DeepStorage then. Have a good one šŸ™‚
i
Thank you, you do the same Thomas!
v
it is a kill tasks job to remove unused segments from deep storage. You said 
And when I issue a kill task to remove them from DeepStorage, nothing happens.
. there is something going wrong there. How are you issuing the kill task?
t
Hello Vadim, Iā€™m issuing it from the console
v
does this datasource have streaming ingestion?
t
Yup
v
it was fixed here: https://github.com/apache/druid/pull/13431 and will be released in the upcoming 25.0 release
the issue is that the console make a kill task with interval 
1000/3000
(so all time). Since that interval overlaps with your streaming ingestion, those tasks just sit there pending waiting for streaming ingestion to stop which never happens
right now the workaround is to issue a kill task yourself with an interval that does not include the future
from the 
Submit JSON task
dialog in the Ingestion view
t
Iā€™ve noticed this issue on the intervals.
Thatā€™s why I issue kill task by hand
Iā€™ve tried to delete on the interval 2021-01-01/2022-01-01 but still no deep storage deletion
v
strange, do you see that kill task get marked as SUCCESS in is status?
t
Yup
And a task duration of 000000
v
Could you please share the task log for that kill task? I would expect to see:
many 
Deleting segment[..]
lines
t
Not at work right now. Will do tomorrow first hour
šŸ™ 1
Thanks a lot Vadim
v
No problem. Would love to get to the bottom of this.
t
Hello Vadim, Iā€™ll try to remove segments from deesptorage using :
Copy code
{
  "type": "kill",
  "dataSource": "beam-realtime",
  "interval": "2022-09-14/2022-09-15"
}
And here are the indexer logs I got :
Copy code
2022-12-16T10:15:20,216 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task[kill_beam-realtime_emdmldij_2022-09-14T00:00:00.000Z_2022-09-15T00:00:00.000Z_2022-12-16T10:15:20.189Z] started.
2022-12-16T10:15:20,216 INFO [[kill_beam-realtime_emdmldij_2022-09-14T00:00:00.000Z_2022-09-15T00:00:00.000Z_2022-12-16T10:15:20.189Z]-threading-task-runner-executor-5] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2022-12-16T10:15:20,269 INFO [threading-task-runner-executor-5] org.apache.druid.indexing.overlord.ThreadingTaskRunner - Removed task directory: /opt/druid-data/task/kill_beam-realtime_emdmldij_2022-09-14T00:00:00.000Z_2022-09-15T00:00:00.000Z_2022-12-16T10:15:20.189Z
2022-12-16T10:15:20,274 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task [kill_beam-realtime_emdmldij_2022-09-14T00:00:00.000Z_2022-09-15T00:00:00.000Z_2022-12-16T10:15:20.189Z] completed with status [SUCCESS].
I donā€™t know if that matters, but Iā€™m using Indexers and not MM
2 Views
Open in Slack
PreviousNext
Powered by