Do we have any solution for query returning missing data when segments are not available on any historical (i.e. historical serving the segment is DOWN/crashed/etc)? i.e. if a segment for T1 is on historical #1, T2 is on historical #2 and query is querying for T1 and T2 but historical #2 crashed and is down at that moment….the query would still run and succeed but only returning data from T1. What I want/thinking is a query can know what segments it needs (from the metadata store) vs what is being served by historicals. If there is a gap between the two set, then alert/fail/etc the query as data needed for this query is missing/not loaded

may be this parameter helps

druid.broker.retryPolicy.numTries

. when it retries it will have a entry broker log so we can have alert on this

I don’t think the query would retry since it doesn’t fail

query won't fail but if broker send a subquery for any segments and if the data from that segment is not available it will retry again . if you have 2 replicas for a segment then it will help if a historical is down so that broker can talk to another historical where that segment is availble

If a segment is not available and is not announced by any historical, then the broker won’t know if it exists and will not attempt to query it though

Hi @Maytas Monsereenusorn, I had worked on something similar to detect segment unavailability in brokers https://github.com/apache/druid/pull/13967. This is still open though.

@Rishabh this is exactly the problem I am speaking of. The Pr also looks very promising. Do you know why is the PR closed? Was there some blocker/concern/etc? Thanks!

I don’t think there were any blockers as such, just that this work got deprioritised and I was not able to pick it up again. cc @Abhishek Agarwal

@Rishabh @Abhishek Agarwal Is there any plan on picking this back up? Is there anything I can do to help?

Hi @Maytas Monsereenusorn, I discussed this with @Karan Kumar as well. We do not have enough bandwidth to work on this full time, so we cannot promise an ETA. However, I will spend sometime to rebase it and address any pending review comments sometime next week. It will be great if you could help out with the review.

Do you need help with the rebase, addressing comments, etc? I can also help pick it up

Thanks, I think it should be fine. I will update.

@Rishabh I reopened the PR. Please let me know when it is ready for review. Thanks!

Hi @Maytas Monsereenusorn, I’ve updated the PR to address the pending feedback. Please feel free to review it. l have identified some additional tasks outlined in the description that I’ll work on in my free cycles. Let me know if you have any questions.