This message was deleted.

Hi Vamsi, if you are trying to backfill historical data from S3, I would recommend doing that first and then setup the Kafka stream ingestion. If you are backfilling after setting up Kafka stream ingestion, please note that a batch ingestion task will fail if it tries to write data for the same time interval which is being written by a stream ingestion task. Druid Lookups can only be based on top of one source.

Thanks Ravin. But, is it even possible to configure a datasource with both ingestion specs? the need for backfill after kafka stream ingestion, is to fix corruption or replay dropped events due to parse errors that upstream fixes. In both cases, cleanup of such segments, and bulk ingest of that interval a possibility?

As long as your batch ingestion is not "replace", I think neither spec will require a lock on the interval, so they should be fine. warning: have not confirmed this first hand.

You should be able re-index older data for the same datasource along with an ongoing real-time ingestion. May I know how frequently are you planning to correct the data?

Frequency is unknown at this time as we are in design phase, but will need a path to backfill bulk fixes when needed.

Got it, so all you need to be careful of is not to backfill bulk fixes for the same time interval which is being written by real-time ingestion. Main problem here would be if there is a lot of late arriving data 🙂

I've definitely seen people do batch backfills on datasources with streaming ingestion. (Sometimes needing to pause the realtime ingestion, if there are locking issues, as Ravin said.)