Hello everyone! I just want to share one thing tha...
# troubleshoot
g
Hello everyone! I just want to share one thing that a noted and could be crucial. After upgrading my DataHub from 0.8.41 to 0.8.44, the user experience was drastically affected. After a few days of trying to debug, I decided to rollback my development instance to 0.8.41 version. Below, I will share the results of the same query, the first in the production environmt with the version 0.8.44 and the second in the development environment with the version 0.8.41. Searching for "order" - version 0.8.41 First try: | 43 requests | 5.6 MB transferred | 5.6 MB resources | Finish: 5.50 s | DOMContentLoaded: 3.82 s | Load: 4.23 s | Average time is 6 seconds to load Searching for "order" - version 0.8.44 First try: | 32 requests | 5.7 MB transferred | 5.7 MB resources | Finish: 34.64 s | DOMContentLoaded: 3.84 s | Load: 4.36 s | Average time is 14 seconds to load Disclaimer: This performance impact has been reported by ALL users of the platform but the above data does not necessarily prove it. I just want to know if anyone else has gone through the same situation. Does anyone know why?
thank you 1
b
hey @gentle-camera-33498 im curious about the scale of datahub you have, roughly how many entities do you have inside?
g
I have about five thousand entities. But, with help, I discovered what was the problem. The Google Cloud DNS was interrupting the GMS's database connections because of a configuration called "TTL(seconds)". Now, everything is working fine! I'm working with the following setup: * Frontend: 1 replica - CPU 0.1 Mi - Mem 1GiB * GMS: 2 replicas - CPU 0.1 Mi - Mem 2 GiB * No standalone consumers * No Actions Framework * Elasticsearch: 3 replicas with 2 master nodes - CPU 0.5 Mi - Mem 4GiB * Kafka and Zookeeper: 1 replica - default resources
b
Was that variable expressed in any helm chart?
g
No. This configuration you set when you create a new record in your domain name list in Google Cloud DNS.
b
Was your sql an external instance?
g
I created domains for my Cloud SQL instances. This TTL parameter kills opened database connections when the permitted period is over.
b
I see... Thanks for sharing!
g
Copy code
Was your sql an external instance?
Yes! I use a Cloud SQL instance to host my MySQL database.
One important thing: My users spend at least 30 minutes daily using the UI. So I changed the minimum number of connections to 5 and limited it to 50 connections, and I set the connection age to 1 hour at least. After these changes, I haven't received any more bug complaints.
b
@big-carpet-38439 maybe you guys would find this useful for troubleshooting.