This message was deleted.
# troubleshooting
s
This message was deleted.
1
w
submitted spec above ^
a
A realtime task ingests data and simultaneously serves it for queries. Once the ingestion duration has elapsed, it waits for the data to be loaded on historicals before stopping so that its data is available for queries. It appears that your segments were created but they were not loaded on the historicals by the coordinator. 1. It'd be helpful if you could check metrics such
segment/assigned/count
during this period to see if segments are being assigned by the coordinator. Coordinator logs would also be helpful 2. You could also check if the historicals are loading the segments if the coordinator has successfully assigned them
It's also possible that both assignment and loading are happening but slowly. You may want to check if 1.
coordinator/global/time
is high 2. replicationThrottleLimit and maxSegmentsInNodeLoadingQueue are too low
g
one additional note: you can read more about how the handoff works here in the docs: https://druid.apache.org/docs/latest/design/architecture.html#indexing-and-handoff
It looks like your tasks are getting stuck on step 4:
If the indexing task is a realtime task, then to ensure data is continuously available for queries, it waits for a Historical process to load the segment.
As @Amatya Avadhanula mentioned, the most common reason for this is if your Coordinator is being slow to assign segments for some reason, or if your Historicals are unable to load them for some reason
If they're never loading them, a common reason is if they cannot contact deep storage. If that's happening you'd see errors in the Historical logs
w
ok ... I'll try to see the things you send here ... thanks
fixed! thanks 🙂
g
🙌
l
if your Historicals are unable to load them for some reason
For future reference, the issue here was the S3 policy missing read permissions for the historicals (but it had write permissions so the tasks actually submitted data) breaking the handoff
1