This message was deleted.

Hi Ondra, welcome 🙂. Re: lookups and Kafka, first take a look at this doc page: https://druid.apache.org/docs/latest/development/extensions-core/kafka-extraction-namespace Then let us know what questions you have based on the information provided. Thanks. John

Hi John, thank you, these extensions seem to be exactly what I was missing. My original goal feels much more achievable now. I still need to figure out a few things but those might easier to verify after starting with the implementation. • Our Kafka record values usually have a few fields (as either JSON or Avro) rather than single value. It would be ideal to load all fields/columns and then refer to a particular filed in SQL. If this is not possible, I can probably extract the field at ingestion time and potentially have multiple lookups sourced from a single topic. • I assume data from compacted topic are treated in SQL as a lookup rather than a table because of major internal differences (different storage, no history etc), however using a table abstraction (like a KTable in Kafka Streams) would still be handy as they could be listed and queried as other tables but still efficient as lookups.

There is a concept called "dimension tables" or "indexed tables" being worked on, that would essentially give you Key + multiple_attributes ... but I don't know if/when that will be making it into the product. Possibly you could ingest into a regular datasource at intervals, following by a replace/update to your target datasource, that could be used for joining at query time ... but my knowledge of streaming ingestion is limited in this area so hopefully someone else can clarify if this or other options would be viable for your use case.

Hmm, limitations for that extension say that

group.id

can't be configured and is set to

UUID.randomUUID().toString()

which won't work with Confluent Cloud and our ACLs. :( Yes, maybe I'll try to ingest into a regular datasource but it might be difficult to deal with deletions (nulls/tombstones) in queries.

When ingest/overwriting a datasource, whatever segment timeline(s) you write in that operation will overshadow the underlying segments ... queries will automatically see the newest data for any given time range. So the easy thing to do here is a full replace ("all" timeline) of the entire datasource ... then the query will pick up only the new data as soon as the job finishes and the new segments are available.