This message was deleted.
# general
s
This message was deleted.
s
Example: Consider a Druid datasource that stores data for the following time range. This data is written by a batch ingestion job.
Copy code
20230101
20230102
20230103
Now, consider a scenario where the new data is written to the source for the following dates
Copy code
20230101
20230103
and deleted for
Copy code
20230102
Now when batch ingestion is run again in
overwrite
mode, Druid datasource will have data for:
Copy code
20230101 (new)
20230102 (old)
20230103 (new)
What we would like is a behavior where if data was ingested specifying the interval as
Copy code
2023-01-01/2023-01-03
then, if there is no data in the source for any date (in this case
20230102
), then the data for the corresponding date is deleted from the Druid datasource.
s
Thanks, @Abhishek Agarwal. It looks like a native batch ingest option only.
a
MSQ uses tombstones by default afaik
oh you probably mean Hadoop ingestion? IMO it shouldn't be a lot of work to add support for Hadoop ingestion