Hi team, we have ingestion lag metric(

ingest/kafka/lag

) this metric will return the lag across all partitions. If we have ingestion replica enabled we take maximum offset consumed across all replica for each partition and use that to calculate lag. But sometime what happen is one task is making good progress but the other replica is not. So, I am planning to add a new metric to expose min lag(take minimum across all replica). So that we can monitor the difference between ingestion task replica. Any opinions? Thanks cc: @Abhishek Agarwal @Amatya Avadhanula @Xavier

why

ingest/kafka/partitionLag

is not enough for you?

If we have replica enabled and even if only 1 replica is making progress we will not see lag for partitions it's consuming from. And there might be a case if the task is consuming from multiple partitions and it makes good progress for one partition and not at all progress for another partition, so this task will continue to hold the data on the indexer till it reaches the task end duration. So for monitoring such cases I want to have this new metrics, for normal cases there should not be much difference between these two metrics.

if it consumes several partitions, then you will get metric per partition imo. the metric above has dimension of partition see here https://druid.apache.org/docs/latest/operations/metrics/ regarding replicas - why it would matter for you? I mean after all only 1 replica will "commit", so as soon as either of them keeping up - you are ok, no?

If you are concerned about all the replicas making progress, you would also have to add taskId as a dimension to the metric

If you are concerned about all the replicas making progress, you would also have to add taskId as a dimension to the metric

I think without taskID also its fine. As, once we know for one of the partition we are not making progress on one of the replica one can check on druid UI to find the respective taskId and Indexer.

How would this metric help with the multiple checkpoint issue?

The multiple checkpoint issue gets triggered when one replica doesn’t make good progress and hold on to the segment for longer time then it hits the maxTotalRows that issue a new checkpoint and it will continue to issue new checkpoints until its handoff the segment. Now we plan to monitor the difference between max and min lag for each partition and based on that we can accordingly set maxTotalRows so that we don’t hit this issue. And we will also get a idea like by how’s the diff vary across ingestion replicas.

Have you tried setting maxTotalRows to a very high value?

Yes, for now we have set the maxTotalRows to very high value. But for monitoring I wanted to add this metric, if you guys think it will be useful.

I think just adding lag min and max lag is not very helpful. It would be nice if you could identify the task from these metrics as well

yeah that also make sense. So instead of per partition we want to expose this metric per ingestion task.