Hi All, Problem : Datahub searches are getting ver...
# troubleshoot
w
Hi All, Problem : Datahub searches are getting very slow Current Datahub is like, We are using Datahub in prod and we loaded around 10k Datasets and have 27 Active users. We are running Datahub pod are with 1 CPU, 4GB and I am seeing memory usage of 90%. So we will have to scale up the no. of pods for GMS along with better memory. Can anyone here have faced search slowness issue and identified the components to scale and how much should they be scaled ?
✅ 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: 1️⃣ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? 2️⃣ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues?
a
this will likely be fixed by allocated more resources to the pod, 1cpu and 4gb will yield pretty poor performance
you could try just scaling cpus initially
w
thanks @astonishing-answer-96712 will try it
I have not given any CPU limit for GMS and I cannot see any CPU utilization > 1 in metrics. Should there be any other factor to scale the GMS pods
a
what version are you on and what’s your deployment environment? a GMS log may also help here to see if something is off- is the frontend itself functioing?
w
Frontend mem usage is <50% we are using AWS - EKS
now I have scaled GMS to 4 replicas with 4G and 8 4 core But it still consumes all the memory and I can see the search latency increasing with time
a
What version are you on? If it’s an older one, this may be a performance issue that’s been fixed since
w
Datahub version :
0.10.1
a
@brainy-tent-14503 may be able to provide some guidance here
b
The jvm is likely configured to consume the memory given since it manages a chunk of its own memory. For the number of users and datasets, 4 replicas is definitely not needed. For search performance, I would like to know the configuration of Elasticsearch, specifically the resources used, number of replicas, etc.
w
Thank you for the reply @brainy-tent-14503 We can close this thread and move to https://datahubspace.slack.com/archives/C029A3M079U/p1681745743054409