It seems there is an availability blip when a segment is moved from realtime to historical. When we run reports at the end of an hour (likely around the time handoff occurs) our numbers are inaccurate. We have to wait until the segments are fully available on historicals before we can trust our reports. Has anyone else experienced this and/or is there a solution?

Hmmm ... are you using concurrent append/replace? In general, the Historical should announce the segment before the ingestion task unannounces it, so there should be no gap. Concurrent is a new feature, though, so possibly a loophole that hasn't been found yet ... Also something to think about is the fact that a reindex task produces a "core set" of segments to replace the existing segments for that time interval ... the Broker should not use the newer reindexed data until the entire core set is marked as available for querying. Also can we assume you have historical segment replicas > 1? Are you using ingestion task replicas? Otherwise I suspect some kind of race condition or Coordinator latency, but this is now beyond my product knowledge so someone else will have to chime in 🙂

We're not using concurrent append. We reindex into a new datasources so no issues with old segments getting overshadowed. Our datasource replica is set to 2 but we don't have ingestion replication.

Do you mean to say that you need to wait for a few minutes for all the new segments to be loaded onto historicals for the other reindexed datasource for it to be query-able correctly? If yes, that is expected behaviour

Hi, it's the source datasource that is initially queryable (realtime) but there are data gaps until it finishes copying to the historicals.

Where do you see the data gaps? In the reindexed datasource?

The reindex job kicks up at around 15 minutes past the hour and reindexes from source for the past hour. There are data gaps. So we have started at around 45 minutes past the hour so there are no data gaps. Ideally this should be able to be done with 2-5 minutes past the hour since there is no lag in ingesting the data from Kafka.

But the gaps in data are in the reindexed source if you reindex early on, right?

Good to knw

Okay, now I understand what you are talking about. So as Amatya has mentioned, other than MSQ with the "REALTIME" flag set, reindex jobs don't look at the real-time segments. Which means it is taking a while, you said well over 15 minutes past the hour in some cases, for the segments to move from ingestion to historical. The delay may be in how the ingestion tasks are cycling. If you have a 1 hour taskDuration in your supervisor spec, and HOUR segment granularity, that does not necessarily mean that the prior hour's data will automatically be pushed to Historical on the hour ... that push will happen only when one of the segment building thresholds is reached -- maxRowsPerSegment, maxTotalRows, taskDuration, or intermediateHandoffPeriod. The tasks don't cycle on the hour, they cycle only when they reach their taskDuration. So if you want to make sure that you "prior hour" segment is published quickly, you may have to set taskduration to something very short, like 10 min, to force an ingestion task rollover which results in a segment publish.