steep-alligator-93593
05/16/2023, 10:31 PMThe node was low on resource: ephemeral-storage. Container datahub-gms was using 8220940Ki, which exceeds its request of 0.
Pod The node had condition: [DiskPressure].
Any ideas on where gms stores it's data? And why this could be happening?steep-alligator-93593
05/17/2023, 2:43 PMbrainy-tent-14503
05/18/2023, 9:04 PMephemeral-storage
is not something that is being set by our helm charts by default. Therefore it is likely inheriting a default from your namespace or something like that. You could populate values for your environment for this resource here. It depends on the activity but probably 1-2Gi should be enough.steep-alligator-93593
05/18/2023, 9:06 PM/
and /etc/hosts
mount paths and both of those are 20gb
, so by nature of the application what is being stored there?aloof-gpu-11378
05/18/2023, 9:12 PM/etc/hosts
, only place I can think of is the logs in /tmp/datahub/logs/gms
steep-alligator-93593
05/18/2023, 9:13 PMsteep-alligator-93593
05/18/2023, 9:14 PMgms-log-cleaner
?steep-alligator-93593
05/18/2023, 9:16 PMaloof-gpu-11378
05/22/2023, 5:37 PMsteep-alligator-93593
05/22/2023, 5:37 PMaloof-gpu-11378
05/22/2023, 5:38 PM-Dlogback.configurationFile=file:/path/to/file
aloof-gpu-11378
05/22/2023, 5:38 PMsteep-alligator-93593
05/22/2023, 5:39 PMsteep-alligator-93593
06/06/2023, 6:12 PMbrainy-tent-14503
06/06/2023, 8:39 PM/tmp
in the pod would work. Alternatively increasing the *node*’s root volume and not using a pvc/pv are both options.steep-alligator-93593
06/06/2023, 9:38 PMsteep-alligator-93593
06/06/2023, 9:38 PMaloof-gpu-11378
06/06/2023, 9:59 PMsteep-alligator-93593
06/07/2023, 1:19 AM2023-06-07 01:18:25.008:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@4f80542f{/,null,STOPPED}{file:///datahub/datahub-gms/bin/war.war}
25
java.lang.IllegalStateException: Parent for temp dir not configured correctly: writeable=false
steep-alligator-93593
06/07/2023, 1:20 AMaloof-gpu-11378
06/07/2023, 2:34 AMsteep-alligator-93593
06/07/2023, 3:36 PMsteep-alligator-93593
06/07/2023, 3:42 PMsteep-alligator-93593
06/07/2023, 5:42 PMbrainy-tent-14503
06/07/2023, 7:47 PMbrainy-tent-14503
06/07/2023, 7:48 PMoverlay
is not specific to the pod, it is specific to the host.brainy-tent-14503
06/07/2023, 7:49 PMdu -sh /
from inside the pod to see what is actually used by the running pod at that mount point /
I believe.steep-alligator-93593
06/07/2023, 7:52 PMbrainy-tent-14503
06/07/2023, 7:56 PMsteep-alligator-93593
06/07/2023, 7:59 PMsteep-alligator-93593
06/07/2023, 7:59 PMsteep-alligator-93593
06/07/2023, 8:11 PM/
path just so I can give all the pods more storage?steep-alligator-93593
06/07/2023, 8:11 PMaloof-gpu-11378
06/07/2023, 8:35 PMflaky-painting-93216
08/11/2023, 12:55 PMsteep-alligator-93593
08/11/2023, 5:02 PM/tmp
directory via the values.yaml
steep-alligator-93593
08/11/2023, 5:03 PMvalues.yaml
below
datahub-gms:
enabled: true
image:
repository: linkedin/datahub-gms
# tag: "v0.10.0 # defaults to .global.datahub.version
resources:
limits:
memory: 3Gi
requests:
cpu: 100m
memory: 2Gi
service:
type: ClusterIP
extraVolumes:
- name: data
persistentVolumeClaim:
claimName: edm-datahub-gms
extraVolumeMounts:
- mountPath: /tmp
name: data
podSecurityContext:
fsGroup: 101
flaky-painting-93216
08/12/2023, 10:59 AM