This message was deleted.

Yes, else the data , when queried, may not be accurate. Can you increase the segment cache ?

Thanks for you quick help, I can increase the cache, but if I got 20T data in druid in the future, then I need to increate the cache to 20T, right?

The question is whether you really need to query all of the time intervals for 20T. That’s why we have retention rules

I am afraid currently I need to cache all the data, as the query is randomly.

Kao-Jun a few things to check -- • If by "cached" you mean loaded onto historical for querying purposes, then this should be based on the disk capacity of your historicals, not the RAM or JVM Heap. What is the total disk capacity of your historicals? • How much "actual" datasource size do you have? In the Druid Console Datasources tab, what does it in the "Size" and "Replicated Size" columns? • In that same tab, what does it say in the "Availability" column for your datasource, both % availability and # of segments? • If you have a large # segments then you should also confirm your segment structure (size and partitioning scheme) is optimal, as you may be able to reduce segment count and increase compression. If you ingested via real-time streaming and did not run compaction, there is a chance that you have significant segment fragmentation due to the real-time ingestion jobs. There are a few things to look at here but relatively easy to spot and clean up.

@宋考俊 Just to add to Brandon and John's comments, and to answer your question. 20T of raw data can be stored in a much smaller representation in Druid. If source is JSON formatted, we usually see about 90% compression, but your mileage will vary based on your raw data format and the data's demographics (distinct values and their distribution in time and across segments). So 20T of raw data could potentially fit into 2T of cache, but you will need enough cache space on the historicals to accommodate the segments that result from the ingestion. It is also normal to have more than 1 replica of each segment on the historicals to provide more parallelism and high availability in the case of loss of a historical or rolling upgrade. So normally it is 2x the compressed size (sometimes more if desirable for higher availability or higher concurrency of queries on the same data).

Thanks so much for your reply. • What is the total disk capacity of your historicals? 600G • The "Total data size" is 1.6T, and the Replicated size is 2.4T • As now I scale the historical to 6 pods, each of them has 500G cache, now it is fully available, and we have 138,745 segements • For compaction, as most of the segments have 5 million rows, I don't think I have much space to compct.

Hi @宋考俊, this is what I got:

If replication is 1 you will need 20T across the 3 historicals, so like 6.66T / historical (20/3)

OK, thanks. Another question is about the query time: I want to query the data using one field:

SELECT * FROM table WHERE iri='xxx'

But it is pretty slow now, it takes 300 seconds to return the result. Now the historical pods are launched with 2 cores and 10G memory, does it make sense to increase the cpu and memory if I want to optimize the query time?

Some suggestions : 1. You should do a range partitioning on

iri

with other commonly filtered dimensions 2. Are you using any emitters like Prometheus ( or Imply Clarity) to determine where the bottleneck is? If segment scanned time is >=250ms and < 500 ms, the historicals are ok in general 3. Broker-wise, you can check the wait time for query and the httpNumThreads available

@宋考俊 Is there a timestamp field in your data? If so, the primary partitioning of the data will be on time and adding a time filter to your query will help to reduce the number of segments to process. Additionally secondary partitioning on

iri

as @Brandon Tan suggests would be ideal in either case, if you are using SQL ingestion, this is a

CLUSTERED BY iri

. It can be specified as

range

partitioning in native batch ingestion under the

partitionsSpec

which will also reduce the segments to process when you use

WHERE iri='xxx'

. Without clustering/partitioning and no time filter, the query would need to process all 138,745 segments. If you have 6 pods * 2 cores = 12 total for the historicals, this means that a maximum of 12 segments will be processing at any one time, and if each segment scan uses 250ms average, that means 138,745/12 = 2890 seconds. Given that you are seeing 300 second response, some of my assumptions are wrong, perhaps you are filtering segments with time or the segment scan time is faster given that the automatic index on

iri

will likely quickly eliminate any segments that don't contain the value. The overhead of opening the segments to inspect them will still be at play though. Let us know how it goes.