This message was deleted.

I was using 4, now I've set it to 2

So it passed the distribution step? I was thinking you were likely running out of memory if you had overcommitted memory.

resources:
        requests:
          cpu: "2"
          memory: 5.7Gi
        limits:
          memory: 12Gi

the node has s-8vcpu-16gb but is shared with other process like historicals, brokers, etc but in theory, when k8s send a kill signal if the node pressured, it send to the entire pod, not just to one container, muhc less just one jvm process

That’s for each MM and you have 4. At the limit it would mean 12x4 48gb. Or am I misunderstanding and each MM runs on a diff node?

I will wait this compaction finishes before changing anything, I tried adding 4 new nodes (dedicated to MM) and having 7 instances, set the max subtask 16 earlier, but it just create one task with 4 subtasks

I see. When the JVM fails, can you capture the core dump?

I don't saw a JVM fail in this case, just this log about it going to shut down

Maybe we’re barking up the wrong tree. That’s an info message. Could this just be a normal finish of the task and the problem was in starting the next phase?

I'll wait this task finishes, then look into separating the ingestion MM and the compactions tasks into different tiers

Sounds like a plan. Good luck. Let us know how it goes.

thank you, I'll keep you all updated!

😎

so, I never used it, but in theory I knew it was possible

Isn’t there also a task type “compact”?

oh, I hope it goes into the default as well

😁

compaction went to the "unzipperman" tier, but I think I was 'by chance', that's wasteful as this task is just doing supervisor and do not run anything itself

True.

the compaction jobs throw some errors while the subtask is doing IO (download the segment)

2023-03-17T21:23:02,627 INFO [ServiceClientFactory-0] org.apache.druid.rpc.ServiceClientImpl - Service [overlord] request [GET <http://10.244.3.35:8088/druid/indexer/v1/task/partial_range_index_generate_GL_ALL_MSG_SERVER_TIME_hcapdifm_2023-03-17T21%3A22%3A54.539Z/reports>] completed.
2023-03-17T21:23:02,628 WARN [qtp1619129136-174] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask - Encountered exception when getting live subtask report for task: partial_range_index_generate_GL_ALL_MSG_SERVER_TIME_hcapdifm_2023-03-17T21:22:54.539Z
java.lang.NullPointerException: null

maybe I need to increase some parameter on the peon. I'm already using

druid.server.http.numThreads=60

(which is a lot, I think, given I have only 2 brokers, each using druid.broker.http.numConnections=10)

Yeah, that should be plenty of http threads. Glad you figured it out. How high did the memory usage get? On the MMs?

when both slots were busy running partial_index_generic_merge it hit 6200mb

So you ended up reducing the peon settings. Hmmm… that would imply that it was over extending the pod resources. The lack of Errors is weird.

now I have a harder question, what's happen to all the segments that was uploaded but the tasks failed? is there any cleanup script or step when any processes go up? I probably have a few thousand files orphan in the deep storage, given this was running and failing for ~ 1 year

I m curious doesn’t deep storage have any life cycle policy if its cloud based?

I've set no life cycle nor versioning, as I expect that druid do all the version control I think DigitalOcean Spaces do not even have any way of doing it

In order to get cleanup on Deep Storage you need to run kill tasks. The druid.coordinator.kill.on=true, does this automatically. It will remove "unused" segments, those that have been dropped (drop retention rule) or overshadowed. I'm not sure about partial files...

I have this option turned off, because I want to keep the data unload for most of the time, if I turn the auto kill, it will remove both the overshadow and the current version of those segments I still did not tested the delete, but I think it should exists an option to issue kill tasks only from the used period, but maybe this is not necessary if the dropExisting in ioConfig during is safe to use (it's marked as beta on https://druid.apache.org/docs/latest/data-management/compaction.html) If this option is safe, and drop only the previous segments version, then the kill task over the load period would not be necessary

maxBytesInMemory

drives intermediate persists. Intermediate persists are local to the MM, they take the in-memory row buffer and write it out locally to disk in segment format. But is this happened during the publishing stage of the tasks? What does your data look like? Do you have a large number of columns? The segment merge step that occurs before publishing can use significant amount of memory. To mitigate, you can set

maxColumnsToMerge

to reduce the memory footprint while merging columns or reduce the

maxRowsPerSegment

so that overall dataset is smaller when publishing.

first they fail with heap, I think it's probability because I was (before Friday) using

-Xms512M -Xmx1024M -XX:MaxDirectMemorySize=1g

for all middlemanagers. I was having issues with compaction not finishing ever (I was assuming the failures of the tasks were always because of them losing their locking, but I was wrong) Then I did a few tests and ended up with two sets of middlemanagers, one for the kafka ingestions and another set to make the compactions, durign this change, I accidentally reduced from 512mb~1GB to a fixed Xms512M. Now I changed back to

-Xms640m -Xmx640m -XX:MaxDirectMemorySize=3g

The max rows was set to 100,000 (probably did not hit in one-task-duration during normal ops) but the maxMemory was set to zero, so it was 1/6 (~ 85mb), when I set maxMemory to 20mb just to check without changing the DirectMem It failed during the final phase, before publishing. The numMergeBuffers=2, sizeBytes=150MiB and numThreads=2 was kept always fixed, using the formula this was theoretically 750mb (where the limit was 1gb when the task was failing with DirectMem errors), I set a big value just because this node is big (16gb) and usually is running with 4 slots at 6GB. Most of the ingestions are very very low volume (like, 1 record a hour), that's why I want so many small workers Now with 20mb every task finished successfully since 5~ hours ago

Sounds like you figured it all out. Nice!