Hello, I am planning to schedule Druid MSQ reindex...
# general
n
Hello, I am planning to schedule Druid MSQ reindexing jobs that uses
OVERWRITE ALL
for a datasource. Simultaneously, I also have DAGs that trigger MSQE job with
OVERWRITE WHERE __time >= TIMESTAMP '{start_time}' AND __time < TIMESTAMP '{end_time}'
. Would this cause any issues on how druid handled multiple ingestion jobs on same datasources resulting in failures?
j
“Overwrite All” will likely lock all time chunks, so both jobs cannot be run at the same time.
n
Does druid automatically keep the other job in waiting or does it fail immediately?
Also, Would this lock be only applicable during the Segment Publishing phase of the MSQ ingestion, no?
j
For two jobs at the same priority level one should wait. However if one job is running with locks and another one with a higher priority wants the lock, the lower priority job will have it's lock revoked and the job will "Fail" ... the task log will indicate that the lock was revoked. You can see this a lot with compaction jobs (low priority) running against intervals where late arrival streaming data (high priority) comes in.