This message was deleted.

could you share a screenshot of the

Services

tab?

hi @Vadim please find it below its showing historical as 100% but My PVC have enough storage

your historicals are full - that is the issue

*historical*:

## If false, historical will not be installed

##

*enabled*: true

*name*: historical

*replicaCount*: 2

*port*: 8083

*serviceType*: ClusterIP

# The Historical will use any available free system memory (i.e., memory not used by the Historical JVM and heap/direct memory buffers or other processes on the system) for memory-mapping of segments on disk. For better query performance, you will want to ensure a good (free system memory / total size of all druid.segmentCache.locations) ratio so that a greater proportion of segments can be kept in memory.

*config*:

*DRUID_XMX*: 2048m # Heap: (0.5GiB * number of CPU cores) + (2 * total size of lookup maps) + druid.cache.sizeInBytes

*DRUID_XMS*: 2048m

*DRUID_MAXDIRECTMEMORYSIZE*: 2000m # (druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes)

*druid_processing_buffer_sizeBytes*: '100000000'

*druid_processing_numMergeBuffers*: 2 # a 1:4 ratio of merge buffers to processing threads is a reasonable choice for general use.

*druid_processing_numThreads*: 7

This is what my hostorical conf is

Hi @Anant Sharma although you may have enough disk space in your PVC it is only finding 1 GB storage. That is why your historicals are running out of space. You need to configure

druid.server.maxSize

and

druid.segmentCache.locations

appropriately to fix this issue. You can find more information about these configurations here: https://druid.apache.org/docs/latest/configuration/index.html#historical-general-configuration

It is not clear to me that you can configure those settings in k8s. There should be a portion of the file for historicals:

runtime.properties: |
  druid.processing.numMergeBuffers=2
  druid.segmentCache.locations=[{\"path\":\"var/druid/data/segments\",\"maxSize\":"${HISTORICAL_VOLUME_SIZE}"}]

Hi @David McHealy you need to change the maxSize. What is it currently?

I don't set that setting. It is inferred from my setting for segmentCache.locations (it totals them).

Hmm in that case you probably need to check the templates and find out what is defining

${HISTORICAL_VOLUME_SIZE}

As I said, that is an environment variable I set. You can just not use that and set your size into the config if you want.

I see. In that case why is Historical size showing up as 1 GB?

No idea. I assume it is writing to the root disk space that k8s makes available to your pod. You definitely do not want to use that. It has to point to the location you are mounting a volume.

Sorry @David McHealy I guess there is a misunderstanding. Maybe you are trying to answer the original question that @Anant Sharma asked and I misunderstood the answer thinking that both of you work in the same team

Sorry I thought you were the original question asker.

Lol 🙂

I personally set an env variable in my ci pipeline that I can pull the historical segment size from so that I can change it easily without changing the config, so that I can use different values between staging and prod environments without changing the config file.

Makes sense. I used to run a Druid Kubernetes cluster on spot instances. It was fun.

I think I remember seeing an issue where they started defaulting it because people ran into that issue a lot.

Yea I used these almost 2 years ago. The official helm charts were deprecated around that time. After Druid 0.20. So, makes sense that it may have changed