Hi Druid team, I Started working with Druid. Ingesting the data through Kafka streaming from the denormalized data table. how can I handle the updates on the existing data in Druid tried reingesting updated data from kafka streaming but on Druid, it is becoming a duplication of the records. What is the best approach to handling the updates on data in Druid? can anyone help me on this

Hi Mustafa, Updated to Druid data can be done in two ways that I know of: • insert/overwrite jobs that copy/modify data over itself, generating new segments in the process. (DOC HERE) • Lookups, which are in-memory key-value pairs that can be updated in real-time via a Kafka stream. (DOC HERE)

Hi John, Thanks for replying to my query. I have tried updating the Task API with

appendToExisting: false

but all the data is overwriting instead of updating only the specific column values. If I try with reingesting the data from kafka stream getting duplicate records. And also tried Lookups with Kafka getting the following error on druid console

druidException / Object 'sales_data_lookup' not found within 'lookup' (line [1], column [22])

Kindly let me know if I am missing anything

@Mustafa,you may want to see if

LATEST_BY

might work for you https://druid.apache.org/docs/latest/tutorials/tutorial-latest-by

Thanks @Charles Smith it worked for my case. Also, may I know if we can remove the old records for those already new duplicate records with updated values ?

You could reindex based upon your needs & use case if you don't need to keep those records.