Does Druid support read locks? My scenario is I have a supervisor that ingests from kafka (hourly tasks). As soon as data is published, I want to ingest that druid datasource into a new druid datasource. Today, I have to estimate when the realtime ingestion task will be done (i.e. have written everything to historical) and leave enough of a time gap to ensure that everything is written. This is error prone. But if we could somehow tell an ingest task to "not start unless your source datasource has all its data" then our 2nd ingest could be significantly improved.

Does your Kafka topic not get data continuously? When new data is arriving continuously, it's hard to define what "everything is written" means

When the 2nd job (ingesting from druid) runs, it does not get the most recent hour (or so) of data because the kafka ingestion task hasn't handed off its data to a historical yet.

This is expected, although I'm wondering what you want, and how you want to define "everything is written" to the source datasource

If the __time value is coming in reliably in chronological order, then I would think once the "prior time interval" lock is released by the ingestion tasks it should not be locked again by them. If so, then I believe kill tasks acquire locks at a very low priority, so am thinking you could test for the lock by submitting a kill task on that interval periodically to see if it runs or goes into a WAITING state. If it waits, then ingestion still has the lock. If it runs it will likely be a benign run that removes no segments. Or is there a better way to test for the lock?

Basically we're doing an ETL and want a way to ensure that the data exists before we kick up the ETL. Running a Druid re-index job has the limitation that realtime data is excluded.

Right ... so as long as there is no interval lock, that means there is no real-time data. Alternatively you could also query the sys.segments table where is_realtime = true.

We have the same requirement. We want to roll up the data from source Druid table (which is ingested in realtime from Kafka) into a daily aggregated table. But in our case, some of the data could have historical timestamps (few hours old) and this leads to issues as those time chunks are then locked for until the Kafka task completes. Also, as the kafka tasks don't run on time boundaries, multiple time chunks could be locked.

If you are having issues with late arrival data locking older time intervals and blocking further batch operations from executing, I can think of a couple of ways to help remediate this: • shorten the taskDuration of the streaming ingestion so the tasks rollover more quickly and release those locks • divert the late arrival data to a different stream, ingest into a side table, and then incorporate that side table data into the main table with the batch job you run on that interval.

if you do the ETL from table1 to table2 with SQL, you can use

segmentSource: realtime

to include realtime data