Hi All, Has any one tried ingesting historical data in druid while the real time ingestion is going on. We took an approach of generating the csv files upload it to GCS and then use batch ingestion. But it takes time and slow when we have huge data set. Wondering if there are some other approaches people has tried. I was thinking if we have a way to create segment directly and tell druid to load these segments ?

The only issues I have seen here is when: • batch and real-time are contending for the same time intervals ... in which case the new concurrent locking feature should help here. • your segment granularity is fine enough that your batch job is opening up a large number of time intervals. Can you provide more specifics on your case? # records, # time intervals, etc?

it varies,. • 6 month data worth 500-600GB data. In terms of records like around 100 to 500 millions records range. • 3 month data worth 500-600GB data. In terms of records like around 100 to 500 millions records range. • 1 year data worth 500-600GB data. In terms of records like around 100 to 500 millions records range. Generally we do not overlap with the real time data range, so we are okay with that.