https://datahubproject.io logo
Join Slack
Powered by
gms service is not starting. it starts and then ex...
# ingestion
f
gms service is not starting. it starts and then exits bxxxxxxx8  linkedin/datahub-gms:head                  "/bin/sh -c /datahub…"  30 minutes ago  Exited (255) 7 seconds ago                                                     datahub-gms b
e
Do you see anything in it’s logs?
f
Dont see any error but can see this
044251.878 [qtp544724190-21] INFO c.l.r.s.c.ResourceMethodConfigProviderImpl - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied: . = 0 044251.899 [qtp544724190-21] INFO c.l.r.s.c.ResourceMethodConfigProviderImpl - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied: . = 0 044251.903 [qtp544724190-21] INFO c.l.r.s.c.ResourceMethodConfigProviderImpl - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied: . = 0 2021/07/28 044252 Command exited with error: signal: killed
m
@fresh-fish-73471 this could indicate just running out of memory on your container
f
Hi @mammoth-bear-12532 it is happening whenever I am trying to ingest or run frontend UI
m
@fresh-fish-73471 are you running this locally on your laptop?
f
Its on EC2 machine
m
Running the whole stack on a single machine?
Or using k8s
f
also one more thing I noticed mysql kafka and elastic search also are in exited state
When I checked the logs for mysql
It says some communication error
let me ping the exact error
m
Are you running with docker or k8s
f
Running with docker
m
Ok, in that case first try increasing the amount of mem allocated to your docker installation
Go for 8GB
Do you know how to set that?
f
No
in env file
java_opts??
I tried setting up in datahub-gms/env/docker.env file to 1024M
It was earlier 256 M
m
How are you checking that the other containers have exited
Using command line: docker container ls ?
f
docker ps -a
m
Ok
You can run “docker info” I think to check the overall settings of your docker env
f
Total Memory: 30.96GiB
is what I can see when I do docker info
m
ok
that seems plenty
right so for GMS, I would bump the mem settings in the docker.env as you found out
JAVA_OPTS=-Xms256m -Xmx256m
-> 
JAVA_OPTS=-Xms1024m -Xmx2048m
f
but that also didnt help i my case
m
is mysql also exiting?
I assume you have looked at the logs? 
docker logs datahub-gms
f
yes I checked the logs by docker logs gms_container_id
b
is there any update on this? i got the same issue
i think the problem lies in mysql
mysql aborted the connection then 
datahub-gms
exited
m
@breezy-camera-11182 @fresh-fish-73471 we are quite convinced that this is happening due to the extreme memory diet that we have put gms on. @big-carpet-38439 will be raising a PR soon to raise that, which should alleviate this problem.
b
ah correct, after i increase the 
JAVA_OPTS
and 
mem_limit
the 
datahub-gms
seems fine
on normal circumstance i think it need around 1GB
b
I raised to 1gb, thanks for verifying Ali!
3 Views
Open in Slack
PreviousNext
Powered by