This message was deleted.

I will suggest moving to http based task assignment strategy that is used by default in 26.0.0. Task assignment can be slower in zookeeper and we usually see HTTP behaving better than zookeeper. If you still see issues, please report a github bug with thread dumps of Overlord process when it gets into such a state.

Did you turn on auto pruning of tasks? We usually never keep tasks older than 24 hours.

Hi @Didip Kerabat I have not done any change related to auto pruning, is there a setting for this?

How long this slow utilization lasts for? Also, what version are you on?

we are on druid 25.0.0. This lasts for ~4mins

what's the metadata store db connection limit on Overlord? I will suggest increasing that. Otherwise, if the timing is predictable, then you can thread dumps on Overlord leader machine.

Did you turn on auto pruning of tasks?

Are you referring to the recentlyFinishedThreshold parameter?

These 3 settings in overlord.properties. For example:

druid.indexer.logs.kill.enabled={{ env("DRUIDOVERLORD_INDEXER_LOGS_KILL_ENABLED", "true") }}

# This is in milliseconds
# 7200000 = 2 hours
# Number of milliseconds of delay between successive executions of auto kill run.
druid.indexer.logs.kill.delay={{ env("DRUIDOVERLORD_INDEXER_LOGS_KILL_DELAY", "7200000") }}

# This is in milliseconds
# 86400000 = 24 hours
druid.indexer.logs.kill.durationToRetain={{ env("DRUIDOVERLORD_INDEXER_LOGS_KILL_DURATIONTORETAIN", "86400000") }}

HI @Abhishek Agarwal @Didip Kerabat I found the root cause of overlord high cpu usage. we have 2 datasources for which coordinator issued kill tasks are failing every 2 hours causing high overlord cpu usage for 5 mins till this kill task times out exception seen in logs

java.nio.file.FileSystemException: /mnt/data/task/workerTaskManagerTmp/<temp_task_folder_name>: File name too long
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]

if I submit kill task manually it passes as the prefix is

kill_*

instead of

coordinator-issued_kill_*

as it is 18 characters less looks like linux file system (ext4 in our case for middle manager instances as i checked ) is having 255 chars limit for file name. In druid is there a way to change this name? can we use uuid for creating folders instead of taskid as task id is only needed in metadata. Thanks in advance.

can't you increase that char limit? Seems low to me. FWIW, first time I am seeing something like this

yes very weired issue. here are few links i found. https://serverfault.com/questions/9546/filename-length-limits-on-linux https://unix.stackexchange.com/questions/721339/what-would-happen-if-i-change-path-max-and-name-max-in-limits-h

@Abhishek Agarwal task id is 255 chars max. Datasource is a part of task id ( in addition to other fields). Datasource is also 255 max. The task_id PK should ideally be a UUID and the time, interval etc should be separate columns. Thoughts? We are happy to contribute this change but I am not sure if this will break something on the compatibility, old scripts for some people etc. You could generally hide this with a config for compatibility but then this is to do with schema so that gets complicated too!

it shouldn't impact compatibility in any way. whats the full task id btw?

@Abhishek Agarwal Druid joins fields here:

static String newTaskId(
      @Nullable String idPrefix,
      String idSuffix,
      DateTime now,
      String typeName,
      String dataSource,
      @Nullable Interval interval
  )

Then there is also "coordinator-issued" and a UUID that gets in somewhere else. Here is an example ( datasource name removed)

coordinator-issued_kill_<DATASOURCE_NAME>_kbhnnlld_2023-05-21T00:00:00.000Z_2023-06-19T00:00:00.000Z_2023-06-21T05:46:17.991Z.599f0ca8-254a-42a7-a702-8f47ecfcbe1b

Then the trouble is not all the fields joined are part of columns in the task table. So have to add columns perhaps to ensure all info remains with tasks. Hopefully we dont parse id to get interval, time of task issues etc

whats the length of datasource name? In case, you can't share the actual name

@Abhishek Agarwal our datasource name is around 105 chars.

that information should be present in spec

255 is the default limit in linux.

I guess you could add the code to truncate if the length exceeds 255