This message was deleted.

With real time ingestion what’s the default time range? hourly?

Yes correct

So some of your jobs appeared within the same hourly interval and this is a deal breaker to you? You cannot have it consumed a little later?

Yes correct

hm… then forceTimeChunkLock = true is the only way. We almost tried it, but we decided that having our hourly job to be consumed slightly later is not a big deal.

yes I realize it's the only way, that's why I was asking for people that had some experience using it before I try it in production

Depending on how often you are dealing with the late-arriving data, perhaps just adding retry logic to the thing scheduling your append jobs will guarantee eventual success? Additionally, if you are using the new SQL-based batch ingestion for the append, it will use a sharable lock for the batch append jobs. You can have your realtime ingestion also use a sharable lock by setting the context flag

useSharedLock

on your supervisor. This should allow for both jobs to run together in parallel. Note, that you should only go this route if you are using the SQL-based batch ingestion. Trying to use it with the JSON-based native batch ingestion has lots of caveats for when it does and does not work. Note, when you switch your supervisor to use the shared lock, you should pause it and let the realtime tasks drain before starting it back up again with the shared lock. If you just set the config and restart the supervisor, your initial tasks will fail as they won't be able to get a lock because of the previously running tasks.

@Eric Tschetter I just tested useSharedLock + SQL-based ingestion on my dev environment and it seems to fit what I need perfectly, thanks a lot! I'm assuming it's not in the documents because of the caveats with other ingestion methods?

Yes, that's why it's undocumented. Especially because the specific configuration of things that make it actually work for non-SQL-based ingestion is just too complex for anybody to understand. We have stuff in the works to just eliminate locking for appending altogether, which will eliminate the config and make it "just work" without caveats. So, that's the path forward.

great, much appreciated!

Something to be aware of with any sort of setup that has a lot of late-arriving data: it's important to eventually compact that data back together. You can do it with auto-compaction or if you. have a scheduler externally, you can schedule things with that. Those jobs can be pre-empted from locks sometimes too and that's okay as long as the scheduling will retry and there's eventually enough of a clearing for things to actually complete.

that's actually what lead me to needing to add this data, I'm appending late-arriving events that I'm putting to the side while running re-indexing compactions on the previous day, we receive certain types of events that need to be set to the timestamp that we're passed and they come in frequently, and it's client-facing so we couldn't wait too long to ingest the late data that came in during compaction

oh, is there not a natural delay where the late-arriving stuff would maybe become more rare?

there is but it depends on a bunch of external systems and because this is reporting data being put off to the side, we can't be waiting 1 week+ for it to become visible to the client

oh, I wasn't meaning delay the data being released

well we decided to compact ASAP because we're using range partitioning to limit the amount of data druid uses when our clients use the UI - we're using Turnilo as our frontend and we built in a query filter using cookies to append "where client_id = ?" to every single query they run, so converting from realtime partitions to range partitions has a good amount of benefit for us, especially when it's a client with lower traffic

Ahhh, that makes sense

but I appreciate all the insight

you'll like it when we finish the support for compaction and append at the same time then

haha that's music to my ears (eyes?)