Hi everyone,
I have a requirement as follows... Please let me know is it possible to do ?
I am using kafka streaming ingestion spec to ingest the data to Druid-datasource-1 in real-time & as a raw-data.
Now, from above datasource-1 -> I want to do parallel real-time ingestion to datasource-2 [near-real-time] with ROLLUP of data within the Druid.
Is it possible to do using Index-Parallel / Re-Indexing - batch process option ? Or any other way to do so ?
b
Bibek Sahoo
10/01/2024, 11:11 AM
Yes you can do Re-index as a batch job, How are you planning to run, is it like you will submit the job on your own,
You can enable autocompaction with rollup, that will do the same thing without manual intervention.
a
Ashok Kumar Ragupathi
10/01/2024, 11:25 AM
How to do using Re-Index in real-time ingestion with roll-up to the datasource-2 whenever datasource-1 new data ingestion done via kafka ? Is it require any scheduling process? after writing re-index spec?
Ashok Kumar Ragupathi
10/01/2024, 11:27 AM
Bco'z I seeing in Re-Indexing ingestion time-interval is mandate. But I want to enable real-time ingestion using index-parallel / re-indexing ?
b
Bibek Sahoo
10/01/2024, 11:33 AM
Thats not possible. reindex is batch only, you need to run in batch mode, Else the autocompaction is the other solution where you can enable rollup and the data will be rolled up upon arrival on the realtime task.