This message was deleted.

Since the partitioning is DAY, when updating say 120 records for a given item_id, we would be re-ingesting data for at most 120 days (as opposed to 10 * 365 days). Would this level of re-ingestion still not be manageable? How many records do you typically have per day?

Yeah, 120 was a wrong number. The point though I was trying to make was that if we were to update data for a single item that has 1 record/per month/per year, then today we have no other option but to re-ingest the entire data. Some of our datasources run into trillions of rows, so this gets very expensive. I guess what I am really looking for here is the capability to have row level updates (delete previous row + insert new row). Or possibly some kind of row level shadowing mechanism where the latest version for a row (identified by a unique identifier) will be returned when there are multiple versions for a row present in the segment. Both obviously are big undertakings. so any crafty solutions with existing ingestion methods would be nice.

Hi Samarth, you can ingest/overwrite at a finer granularity than the existing segments ... this creates what is called "overshadowing", and Druid automatically takes care of seeing only the overwrite. This will allow you to do the updates more surgically, reingesting fewer of the rows that you aren't updating. I wrote up a small doc on it ... let me know if this makes sense for you. I also have a short video that I recorded that I could point you to ...

Interesting idea, @John Kowtko. Let me read up.