Hello, my team and I started to experience a worry...
# troubleshoot
q
Hello, my team and I started to experience a worrying problem with GMS, we have a very particular load for GMS which consists of around 1000 GraphQL requests to get dataset details and each http request is followed by a MCE emitted to the Kafka queue changing dataset which we searched for. For some of the requests an ingress in front of GMS (nginx) returned 502 Bad Gateway status and we believe GMS simply did not accept connections. The problem does not seem to be related to the amount of gms instances we run. Overall load as reported by prometheus at 2m rate was 10 req/s so not much. My question is - is there any way to control GMS behavior under load? Some parameters we could try to tweak?
I may add that connections seem to be consistently dropped after 62-63ms, so it indicates some hard limitation
a
Hi Piotr, you wouldn’t happen to be using AWS, would you? We have a guide on using a load balancer to help with these issues https://datahubproject.io/docs/deploy/aws/#expose-endpoints-using-a-load-balancer
Otherwise, a deployment method and DH version would be helpful to know what your options are here
q
Hi @astonishing-answer-96712 thank you for the answer, I can't find any particular information on performance/random connectivity issues in the linked guide, did you mean something in particular from that guide? We do use AWS ELB and nginx ingress controller
a
What version of DH are you running? @dazzling-yak-93039 may be able to speak to this as it’s odd behavior for GMS
q
0.9.6.1
a
CC: @brainy-tent-14503
b
@quick-pizza-8906 - Can you confirm that you mean maybe 62-63
milliseconds
? In other words, the connection are being almost immediately dropped? Or is this meant to be
seconds
and it is similar to a timeout condition?
q
I meant milliseconds indeed @brainy-tent-14503
a
Hmm, nothing in datahub should close connections like that. Even an immediate authentication error should be returning 4xx http response and not simply dropping connection.
q
Could CPU throttling of a pod cause this? It might be the fault here is on ingress controller side but we try to explore every possibility. We notice 1% cpu throttling as show by kube-metrics on pod around the time connections are dropped - normally I would ignore this as in the past such throttling didn't cause any problems for other services I have been running...
a
I doubt the CPU usage would be a factor unless it was getting a ton of traffic. With connection drops that quick it seems more like a network firewall/security group, ip deny list, or something like this. Another possibility is there is a misconfiguration in the ingress. In this case you’ve mentioned using both nginx and the ALB. I am not sure why nginx is involved when the
aws-load-balancer-controller
uses the ingress to configure the ALB directly. I would add inspect the nginx configuration for errors/misconfiguration or simply not use it.