This message was deleted.

bigger better, but 300TB on less than 1TB (30x32) sounds slow

The answer is influenced by what you’re looking to focus on and the details of the ingest/data sources - “better” is one of those words that can mean something different to the person answering the question, too, so you’ll need to clarify to get a more precise response.

It is preferable to pick instances for the historicals with higher vCPU count. Please be aware that you should probably not keep the segment-cache full (300TB over 30*10TB OR 60*5TB so 100%) as it might lead to issues with the Master. I would keep it 80% Full

Thank you for your response. Can you please explain why 80%? I need to search through all the data quickly, so I would like to have them 100% in the SSD cache on Historical, without referring to deep storage.

You should keep all your data in the segment-cache but keeping the historicals full will cause a host of issues including ingestion failures if not a full system outage. Leaving it around 80% full is a good enough balance where you will use as much of the storage but will leave some space for other processes and keep the cluster functioning. My advice would be to add more historicals so that they end up being about 80% full at most. Also keep in mind, data might grow even if retention is set.

Oh, yes, now I understand, thank you.