This message was deleted.

druid will read the updated data from the present time but if you want to update the data that has already been ingested then you need to reset the supervisor and ingest that data again and use LATEST/EARLIEST in your queries

you might find the "schemaless dimensions" feature to be useful: https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html#inclusions-and-exclusions

The Kafka partitions are defined as "immutable," referring to the fact that a producer can append messages to a partition itself and not change the value for an existing one (append-only log). When the team updates the Kafka, the new Kafka record is added to a Kafka topic partition. The Druid Kafka ingestion task is Kafka consumer. The Kafka consumer receives a new record that includes new data columns. You don't need to consume data from the beginning in this case. But you have to ensure that new data columns are visible for Druid. The Kafka record is schemaless if you are not using Schema Registry, and we can use Druid schemaless dimension. In such cases, you don't need to specify any dimension columns in the Druid ingestion spec, and Druid will treat all columns which are not timestamp a dimension. I think we still need to add flatten spec for nested structures if we have them.