This message was deleted.

What does the log inside MM say? It supposed to say “successfully deleted hdfs://xyz” file

Hi @Didip Kerabat, thank you for your response. The above lines are extract from the MM log itself. I do not see a line stating it is deleting some files from HDFS

Does your YARN show any activity?

I dont have access to my YARN right now, will check and update the thread

It looks like it hasn't even tried to delete anything

Hi @Gian Merlino, correct, it is not deleting at all. I verified, my datasource has a retention of

P3D

, and I see underlying segments in HDFS deep storage from 2021.

the interval in the job

2020-01-01T00:00:00.000Z/2020-12-31T00:00:00.000Z

does not include 2021 -- if you did not do a different job including 2021 as well, perhaps this is why

Sorry, my bad, I should have specified that clearly. I had two separate jobs for 2020 and 2021; and in underlying HDFS I see both 2020 and 2021

ah, got it

You should really debug this inside your YARN. YARN will say if the applicationId that supposed to delete is working or not.

Thanks for your pointers, @Didip Kerabat and @Gian Merlino! I did some investigation from the metadata MySQL table as well, and here are my two observations - 1. I had to specify a smaller time range for clearing the underlying HDFS segments. Since the number of segments in the HDFS directory was too large, even when running

hdfs dfs -ls

, I had to specify extra memory to get the output using

export _JAVA_OPTIONS="-Xmx8g

. Looks like when Druid’s Kill process was also trying to access the HDFS location, it was running into memory issues. When I specified smaller windows, the kill task was able to delete the HDFS segments. 2. The kill process did not delete all the segments in my specified interval, because in the MySQL

druid_tasklocks

table, I could still see the listing of these segments. Some segments dated back to 2020, and 2021. (My datasources’ retention is set to P3D).

is this kill job manual? The automatic kill job supposed to kill 1 day only

Correct, @Didip Kerabat, this kill job was manually triggered.

i see i see. You will run into same problem with compaction as well. In general, you have to be careful with really large time range

Got it, do we have any recommendations on the time range we should use?

We usually do things 1 day at a time, even though the actual time range is large.

Cool! Thanks! 👍

it’s best to automate that using something like airflow because it requires some logic: chunking, retry, backoff, etc. We do this for backfill, kill, and compaction.

makes sense.. Long time back we also had cron scripts that would issue kill tasks, but somehow they got lost in transition from one infra to the other. Thanks again for all your pointers! They really helped!

cross post it to #C030K0Z7S1Z?

Sure