Im looking to assign different run priority to dif...
# general
e
Im looking to assign different run priority to different ingestion tasks. (Meaning, which task gets to run, not related to task locking priority). We're ingesting data in a Lambda fashion: new data is ingested in streaming and then re-ingested after a week with batch. Since the batch re-ingestion has a much more relaxed SLA, we were thinking to let both workload share the same hardware resources, but protect the streaming by giving it higher run priority. But I couldn't find anything about such configuration - does this mechanism exist? I wonder if MM-less can support it, as Kubernetes has these mechanisms built-in.
j
Hi Eyal, the task priority doc page is under Lock Priority: https://druid.apache.org/docs/latest/ingestion/tasks#lock-priority.
e
From reading this chapter about Locking, looks like Task Priority is regarding which segment version wins overshadowing, not which gets more CPU cycles to run. Or did I get that wrong? I’m actually wondering if Kubernetes prioritization can be the answer, but haven’t had the chance to dive into that yet.
b
One idea might to use task affinity, and assign realtime more MMs than batch?