Hi Community, I've noticed that my DataHub is currently utilizing the root volume of the node to run its services. As a result, the volume has run out of spzce. To address this issue, i have set up a new StorageClass and make it as default storage class. However, I'm unsure why DataHub is still using the root volume instead of new StorageClass. I attempted to add the persistence section to the datahub-gms section, but it still utilizes the root volume. Interestingly, I created a simple pod and configured it to use the StorageClass I created. The pod successfully ran the service on the specified storageclass, confirming that the storage class is properly configured. Do you have any suggestions on how to resolve this issue? I've attached the screenshot of the settings for your reference. Thanks

The datahub helm charts do not use persistent volumes for anything out of the box. You’d have to update the helm charts to create a persistent volume claim and then mount it at a specific location. For example, for GMS we might want to persist the debug log at

/tmp/datahub/logs/gms

however in general the datahub components do not require persistent storage themselves so ephemeral is fine. The logs at the above location for example are capped at 10GB. Are you experiencing larger usage? I see that you are allocation 10x that amount.

Hi David, The reason I'm attempting to increase the ec2 root volume from 20GB to 100GB is due to encountering the "NodeHasDiskPressure" and "FreeDiskSpaceFailed" message from the EKS node while trying to ingest data from Snowflake. I suspect that the GMS is filling up the storage and requires more space to efficiently process the data...

It might actually be the actions pod if using managed ingestion. GMS is logs are capped, however I have definitely seen the actions pod require 25G or more. You will have to add the pvc yourself or update the helm chart, then mount it using the

extraVolumes

and

extraVolumeMounts

options, as there is no support for persistent volumes in the helm charts at this time. @famous-waitress-64616 - do you happen to know where the cache is located? I couldn’t find any docs indicating whether it was

/tmp

or not, which would be my guess.