Hello team :wave: We are facing an issue related t...
# troubleshooting
i
Hello team 👋 We are facing an issue related to data ingestion and segment unavailability, would appreciate your thoughts on how is this approached for backfill / regular ingestion flows. We have a Hadoop job that does a segment creation and metadata push for ingestion. Currently, the metdata push done brings the segment ingested ONLINE immediately in ideal state but not yet in external state since the servers take a few seconds to few minutes to download all the segments and there is a segment unavailability during that time.
Is the recommended approach to modify the workflow of metadata push from current default implementation to do a metadata push of the segment first in the state of OFFLINE and then verify that it is downloaded / pushed to servers (this is the part I am not sure which API / how I can go about it ) and then bring the segments ONLINE in ideal state.
m
This is expected behavior. Segment is not available until it shows as ONLINE in external view. There is always a slight delay between IDEAL-STATE and EXTERNAL VIEW updates. What is looking at IS in your case and expects segment to be immediately available?
i
Gotcha, some queries that are coming in constantly / continuously over the time period of the newly added segments are marked OFFLINE in EV and the query fails (like partial results with exception of X segments not available) in that slight delay. We have leaned towards not surfacing partial results to clients so far. So that is the case where segment availability comes up urgently. Is that the right way to look at it and wonder how others handle this part? The problem is exacerbated if ingestion of many segments happen over time continuously (over a long backfill period - 4 hours / 6 hours) and the EV has some segment not ONLINE yet over the long period.