This message was deleted.

you are correct in that the base and replica tasks consuming from the same set of partitions will be used to respond to queries. So yes, there is a possibility that one of the two replicas is slightly behind and could therefore return variable results upon repeated queries. If both tasks have access to the same amount of CPU/Memory resources, then this difference should be negligible.

Thx for the answer Sergio. In the case when one task failed and restarted, the consumed kafka offset can be much different. In the this case, the query results can be significant different, right? Is there any mitigation of this inconsistency?

That's a good question. My understanding is that it is not mitigated but I could be wrong. Sounds like an interesting improvement request if that is the case.

This can be an issue for certain usage cases. Think about using Druid SQL to evaluate alerting conditions every N secs. @Sergio Ferragut, it would great if we can have some of the Druid developer/architect to confirm this design. If this is indeed the case, we would like to hear any ideas to solve this potential issue.

Agreed. I will pursue an answer.

Hi Kai, what you are saying is technically true, there can be a condition where a task is rudely interrupted and needs to catch up, if a query is routed to that task while it is catching up it will appear to go back in time (vs the other task). In reality it is not really a problem that you need to concern yourself with, let me explain:

@Vadim, thx for the detailed explanation. Let me make sure I understand the restored-based config in detail Here, we deploy the druid cluster on the k8. The nodes of k8 be be patched one by one. From the Druid point of view, the MM and all the Kafak ingestion peons can be killed async (just like ctrl-c to kill a program in the terminal). Here, if we turned on restore-based config. When the MM pod restarted on another k8 node, it can retrieve the state information from the database (say Postgres or MySql depending on our config). And the MM would restart the peons from where the kafka offset they ingested at the time they got killed. The question is that how the existing ingested data which can be huge, say 200M depending on the config can be retrieve? They were on the old node which were gone. Wondering if my understanding of this restored-based config is correct here or maybe not right? The reason I bring this up is that indeed this k8 node patch case is something we met in production. The graceful-termination-based approach would help much in this async mode. (Other pre-planned rolling upgrade case seems to be better candidate for the the graceful-termination-based approach.) Would appreciate your feedback here.

when you restart you k8 nodes do the new nodes have access to the disk of the old node?

Actually, I am not totally sure. We use EKS (network volumes for segment caching). This one can be re-attached with the same pod for sure. Let us say if we use EKS for temporary files for ingestion task, would that help?

If your MMs are a stateful set then this will be true.

Right, for statefulset, the MM has identity and EKS would reattach to the same pod. The question is that would the restarted peon pick up from where it left before? Or will the peon just get restarted to ingest from the beginning?

If you re-attach the same local storage and have

druid.indexer.task.restoreTasksOnRestart=true

set then it will pick up where it left before as I understand it

But in general, saying the MM uses instance storage, there is not such guarantee. Then the peon would restart from the beginning, right? So the conclusion is that it would be better to use EKS in this case?

yes, it would also save you processing

Understood, sometime developers may not think too much into operations. Just to confirm the intention of the the two features (or how do we reason about the two approaches) restored-based and graceful-shutdown-based. The restore-based is designed for the async MM/peon failure while the graceful-shutdown is designed for graceful shutdown cases. In the restore-based config, it would be beneficial if the data state-info (partially generated segments) to be available to the restarted peons.

Sounds interesting, but kafka offsets are per partition, and each ingestion task handles a subset of partitions, so this might work if we add all the partition offsets in the context and then have each task recognize its subset of partitions.

Right. In my previous post, I abstract kafka offset as one value. In fact, it is a set of value corresponding to all the partitions for specific peon, from implementation point of viewl

Sounds like an interesting proposal for a change.

I added an issue for improvement based on this discussion. https://github.com/apache/druid/issues/13994. Maybe I will also submit an PR next if this issue turns out big enough for our usage.

I do not understand the mechanism you are proposing who will set the hint?

I do not understand the mechanism you are proposing who will set the hint?

The idea is simple (though may not be the most elegant). When a query is served by a peon, the kafka offsets of the peon is returned. Next time the query is issued again, the previous offsets is also used as the input for the peons. If the serving peon see an offset later then the current offset it is consuming, it should wait (or fail the query) till it catch up. This is the same idea of guaranteeing exactly-once ingestion from Kafka topics. To achieve exactly once, Druid associate the kafka offset with the segment in the meta store database. (Actually, I did not dig the code to confirm this. But this is just a common technic almost all the distributed system use with Kafka).

Oh I see

Also are you saying that you feel like all the mitigation measures outlined above are not enough for your use case?

Right, for some use cases, like alerting, if you can go back in time even once in a while, you own a lot of explaining to the user.

Oh GH is just short for GitHub

Just updated the description a little bit. Basically the same as what I illustrate here. Hope it makes it clearer. Feel free to add your comments in the issue too.