This message was deleted.

Check the coordinator log as well as your historical(s) log

Hey @Sundeep Veeramachaneni, Like Ian mentions, once the ingestion has completed, the segment is loaded in deep storage. It is the coordinator's job to then decide which historical(s) should load the segment, the historicals comply with the coordinator's instructions by loading the segments into their local storage and that's when they become available. So looking for errors/failures in the coordinator and historical logs should give you an idea as to why this is failing.

🙏 Thanks will check

That's the problem. In a k8s deployment

local

is not a good choice. You need to use

s3

or

hdfs

and provide the config params for them. Here's how to set it up for s3: https://druid.apache.org/docs/latest/configuration/index.html#s3-deep-storage

Ahh thanks

local

means use local disk for deep storage which really only works for single server instances where both the MM and Historicals "see" the local disk because they run on the same machine. With clustered deployment, all MMs and Historicals need access to a common deep storage, hence HDFS or S3-compatible object store. In the k8s case using the default helm chart, the MM and historicals are placed in different pods without any shared storage, so that's why

local

does not work.

@Sergio Ferragut Thanks for that detailed explanation

🙌