This message was deleted.

Ran df -h on middle manager

bash-5.1$ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  48.3G     34.3G     14.0G  71% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                    31.4G         0     31.4G   0% /sys/fs/cgroup
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/sda1                48.3G     34.3G     14.0G  71% /etc/hosts
/dev/sda1                48.3G     34.3G     14.0G  71% /dev/termination-log
/dev/sda1                48.3G     34.3G     14.0G  71% /etc/hostname
/dev/sda1                48.3G     34.3G     14.0G  71% /etc/resolv.conf
/dev/sda1                48.3G     34.3G     14.0G  71% /opt/druid/var
/dev/sdc                  3.9G      6.6M      3.8G   0% /opt/druid/var/druid
tmpfs                    56.6G     12.0K     56.6G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    31.4G         0     31.4G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                    31.4G         0     31.4G   0% /proc/scsi
tmpfs                    31.4G         0     31.4G   0% /sys/firmware

what is your deep storage configured to?

Azure storage account

what is -Djava.io.tmpdir in your middle manager config pointing to?

Seems like the ingestion errored out while writing out the logfile which is placed in the baseTaskDir. If everything is set to defaults this should be equivalent to

java.io.tmpdir

. Setting the value to the disk with free space might help. Also did you run the df -h during the ingestion that throwed the error or post the failure?

I ran it post the failure.

if that is in /opt/druid/var/druid then you will have a space issue. Where is the segment cache?

On middle manager segment cache is at

/opt/druid/var/druid/segment-cache

bash-5.1$ cd /opt/druid/var/druid
bash-5.1$ cd segment-cache/
bash-5.1$ ls -lrth
total 0
bash-5.1$ pwd
/opt/druid/var/druid/segment-cache

can you paste the middlemanager runtime properites and jvm config and historical runtime properties and jvm config here?

Sorry, how can I get the runtime configs ?

Are you running on kubernetes?

I think you can get some of the runtime properties from the logs when the middlemanager/historical starts (please mask out any sensitive fields)

Hi. I see this location on middle manager is getting filled while I am running the reindexing.

/dev/sdc                  3.9G      2.7G      1.1G  71% /opt/druid/var/druid

On checking at this path, I saw all the segments are getting downloaded on the middle manager from deep store. And eventually the disk is getting out of memory. As the disk attached to middle manager is only of 4GB

bash-5.1$ pwd
/opt/druid/var/druid/task/index_parallel_rms_logs_query_performance_kkmnpjac_2023-01-06T04:17:09.726Z/work/indexing-tmp/rms_logs_query_performance
bash-5.1$ ls -lrth
total 252K
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T01:00:00.000Z_2022-12-01T02:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T02:00:00.000Z_2022-12-01T03:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T03:00:00.000Z_2022-12-01T04:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T04:00:00.000Z_2022-12-01T05:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T05:00:00.000Z_2022-12-01T06:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T06:00:00.000Z_2022-12-01T07:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:18 2022-12-01T07:00:00.000Z_2022-12-01T08:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:18 2022-12-01T08:00:00.000Z_2022-12-01T09:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:18 2022-12-01T09:00:00.000Z_2022-12-01T10:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:18 2022-12-01T10:00:00.000Z_2022-12-01T11:00:00.000Z

[9:59 AM] Does this means middle manager has to have a disk of size ~total size of the segment ?

You can split the workload across multiple middlemanager using subtasks

you can change

-Djava.io.tmpdir

to some location that has more space