This message was deleted.

If you need to use all 15TB of segments for your iterative testing, then unfortunately you may have to load them each time. If you don't need replicas or need the entire dataset in memory for each test, then you could set replicas to 1 or even 0 for specific time durations to reduce the number of segments that need to be loaded. There may also be parameters that affect overall load speed, I don't know enough about this area of the product currently to comment on it though.

optionally you can lazy load on startup for a historical. Set

druid.segmentCache.lazyLoadOnStart=true

Thank you @John Kowtko and @Arun C. Will definitely try this

https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1686549155518279?thread_ts=1686549155.518279&cid=C0303FDCZEZ has another suggestion in the thread as well.