Hi All, I have question regarding kill tasks locks vs kafka ingestors locks: 1. it seems kill tasks are WAITING due to some locks taken by kafka ingestors, is it ok? something I can do about it? 2. kill task is supposed to remove only unused segments so why it takes any locks? 3. suppose we started to use concurrent append feature (i.e. "useConcurrentLocks": true ) both under compaction & kafka ingestors, do we have option to pass same context for kill tasks?

suppose we started to use concurrent append feature (i.e. "useConcurrentLocks": true ) both under compaction & kafka ingestors, do we have option to pass same context for kill tasks?

Kill tasks cannot run concurrently with ingestion using these locks unfortunately

Hi @Amatya Avadhanula thanks for answers regarding legacy flag of

markAsUnused

you mean that kill task itself can mark segments as unused and kill(aka delete) segments in same "run"?

Yes

if (markAsUnused) {
      numSegmentsMarkedAsUnused = toolbox.getTaskActionClient().submit(
          new MarkSegmentsAsUnusedAction(getDataSource(), getInterval())
      );
      <http://LOG.info|LOG.info>("Marked [%d] segments of datasource[%s] in interval[%s] as unused.",
               numSegmentsMarkedAsUnused, getDataSource(), getInterval());
    } else {
      numSegmentsMarkedAsUnused = 0;
    }

ok, found also this https://github.com/apache/druid/issues/11657

BTW I just saw that there are tasks such as

MoveTask

RestoreTask

and

ArchiveTask

Basically I'm not sure if I want to mark everything unused. Since there are both used and unused segments for recent intervals. Compaction task now runs in parallel to kafka ingestors, using concurrent locks. However multiple compactions create additional load on deep storage and I was trying to run kill tasks to remove old (already compacted) segments...

If you are using concurrent append and replace, please do not manually clean deep storage locations

yes, i don't want to do it

Compaction does not delete compacted segments. I think one way would be to try adding a parameter to kill tasks so that they do not take locks. Or perhaps allow them to use APPEND locks (I'm not sure if this is a good idea)

i mean dropExisting: true from here https://druid.apache.org/docs/latest/data-management/manual-compaction#compaction-io-configuration

I think that is for creating tombstones for index_parallel / compact tasks

I would think Kill Tasks require time interval locks in case you try to re-mark those segments as used again (e.g. retention policy change) while the kill task is running ... would that be the case?

Marking segments as used with the coordinator apis may not check for task locks as far as I know. I'm not 100% sure

@John Kowtko no, not doing policy changes or re-marking as used. My intention is only to remove compacted segments. i'll post later payload

Which parameters specifically?

@Amatya Avadhanula I mean what you suggested above

I think one way would be to try adding a parameter to kill tasks so that they do not take locks.

Ah, no. The change that I suggested would require code changes

oh, i see

You would have to override the

isReady

method for

KillUnusedSegmentsTask

to use

APPEND

locks instead of

EXCLUSIVE

@Amatya Avadhanula regarding your comment of

If you are using concurrent append and replace, please do not manually clean deep storage locations
Different segments may reuse the same file in deep storage

does segment table contains data that can help me to analyzer which file is used and which file is already not used by any segment? I see payload that contains

"path": "<hdfs://taz-il/production/druid_deep_storage/sp_campaigns_realtime_aggregation_by_request_time/20240413T000000.000Z_20240413T010000.000Z/2024-04-13T00_00_01.528Z/7_f908dc30-38eb-4489-b0f4-21d6a4f14fa9_index.zip>"

You would have to scan the entire segments table for this

can you please elaborate about

I recall that you said that you don't mark data as used later. In that case the kill task can handle load spec duplication

I'm interested in contributing 🙂 I'm currently working on something else - https://github.com/apache/druid/pull/16266 (maybe it's less important)

I mean that you don't intend to mark unused segments as used after they have been marked by the coordinator

yes, true that

Let's say you have segments S1 and a deep storage location D such that S1 -> D

yes

Yes

and improve kill task

Yep

ok, understood

https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1713193369135799?thread_ts=1713176533.244569&cid=C0303FDCZEZ Should be simple enough to try out locally

opened this issue, will wait for some comments and then will start work on it https://github.com/apache/druid/issues/16361

Thanks for the issue, and the PR! Will take a look soon

thanks, will address them shortly

I'll take a look soon