This message was deleted.

Iike is there any manager for kafka offset

do you mean the kafka supervisor? you should be able to see it on the ingestion tab of the web console. click on whatever supervisor you want and select details

Ah thanks! I just curious how druid manage offset internally in code level. So I am looking in druid source code and I wonder if there is kafka offset manager or something that handle. Like new Peon Task when pop how does new Peon Task know offset,

The offset is stored in the druid_dataSources table.

Oh I see, Thanks @Saydul Bashar would you recommend me what class has a responsibility of it? Really thank you Cuz I want to understand how druid manage offset that well.

Hi @추호관 you are welcome. I am glad that I could help. I will have a look at the code. I haven't had to look into it for a while.

i don't think it will help look at the table directly, because it's saved as binary I don't think it's all in one class, but the main logic must be in the segment publisher, not in the supervisor, I think

Thanks I will look into that! And if there is any updates on information it would be pleasure to know that

It would be cool if users can manipulate the offset from the UI, @Vadim 😝

we need an API for that first 🙂

do people really think in terms of offsets? What is the usecase here? Just curious

one (niche but perhaps meaningful) use case would be having a second druid cluster pick up ingestion exactly where a first druid cluster left off

I agree for that. I scale out and scale up druid clusters sometimes. (excluding equipment or Including equipment or move to another cluster). So sometimes I want to move offsets manually. And situation when changing kafka ingestion to druid (like flink, kstreams to druid)

Moving offsets manually can be done in the db by updating the druid_dataSources table where the offset is stored. An API will be easier though.

Ahhh I have seen it in metadata storage(in my case mysql)

Yea MySql is my most favourite metadata storage as well. So, you can just bump it up there if you want. I have done it in the past on production systems. Works like a charm. If you know some of the tables that has key information in the metadata table, backups, recovery and moving offsets becomes a lot easier.

yup yup, it’s in the db but an API would make life easier. Any time whenever there’s a “gap” in the stream, you want to manipulate offset yourself. Agree with what Gian said 100%. There are some situations where earliest and latest are not what users want.

Agreed